# Sentiment Classification & How To "Frame Problems" for a Neural Network

### What You Should Already Know

• neural networks, forward and back-propagation
• mean squared error
• and train/test splits

### Where to Get Help if You Need it

• Re-watch previous Udacity Lectures
• Leverage the recommended Course Reading Material - Grokking Deep Learning (40% Off: traskud17)
• Shoot me a tweet @iamtrask

### Tutorial Outline:

• Intro: The Importance of "Framing a Problem"
• Curate a Dataset
• Developing a "Predictive Theory"
• PROJECT 1: Quick Theory Validation
• Transforming Text to Numbers
• PROJECT 2: Creating the Input/Output Data
• Putting it all together in a Neural Network
• PROJECT 3: Building our Neural Network
• Understanding Neural Noise
• PROJECT 4: Making Learning Faster by Reducing Noise
• Analyzing Inefficiencies in our Network
• PROJECT 5: Making our Network Train and Run Faster
• Further Noise Reduction
• PROJECT 6: Reducing Noise by Strategically Reducing the Vocabulary
• Analysis: What's going on in the weights?

# Lesson: Curate a Dataset

``````

In [1]:

def pretty_print_review_and_label(i):
print(labels[i] + "\t:\t" + reviews[i][:80] + "...")

g = open('reviews.txt','r') # What we know!
g.close()

g = open('labels.txt','r') # What we WANT to know!
g.close()

``````
``````

In [2]:

len(reviews)

``````
``````

Out[2]:

25000

``````
``````

In [3]:

reviews[0]

``````
``````

Out[3]:

'bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers  . the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students . when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled . . . . . . . . . at . . . . . . . . . . high . a classic line inspector i  m here to sack one of your teachers . student welcome to bromwell high . i expect that many adults of my age think that bromwell high is far fetched . what a pity that it isn  t   '

``````
``````

In [4]:

labels[0]

``````
``````

Out[4]:

'POSITIVE'

``````

# Lesson: Develop a Predictive Theory

``````

In [5]:

print("labels.txt \t : \t reviews.txt\n")
pretty_print_review_and_label(2137)
pretty_print_review_and_label(12816)
pretty_print_review_and_label(6267)
pretty_print_review_and_label(21934)
pretty_print_review_and_label(5297)
pretty_print_review_and_label(4998)

``````
``````

labels.txt 	 : 	 reviews.txt

NEGATIVE	:	this movie is terrible but it has some good effects .  ...
POSITIVE	:	adrian pasdar is excellent is this film . he makes a fascinating woman .  ...
NEGATIVE	:	comment this movie is impossible . is terrible  very improbable  bad interpretat...
POSITIVE	:	excellent episode movie ala pulp fiction .  days   suicides . it doesnt get more...
NEGATIVE	:	if you haven  t seen this  it  s terrible . it is pure trash . i saw this about ...
POSITIVE	:	this schiffer guy is a real genius  the movie is of excellent quality and both e...

``````

# Project 1: Quick Theory Validation

``````

In [6]:

from collections import Counter
import numpy as np

``````
``````

In [7]:

positive_counts = Counter()
negative_counts = Counter()
total_counts = Counter()

``````
``````

In [8]:

for i in range(len(reviews)):
if(labels[i] == 'POSITIVE'):
for word in reviews[i].split(" "):
positive_counts[word] += 1
total_counts[word] += 1
else:
for word in reviews[i].split(" "):
negative_counts[word] += 1
total_counts[word] += 1

``````
``````

In [9]:

positive_counts.most_common()[:15]

``````
``````

Out[9]:

[('', 550468),
('the', 173324),
('.', 159654),
('and', 89722),
('a', 83688),
('of', 76855),
('to', 66746),
('is', 57245),
('in', 50215),
('br', 49235),
('it', 48025),
('i', 40743),
('that', 35630),
('this', 35080),
('s', 33815)]

``````
``````

In [10]:

pos_neg_ratios = Counter()

for term,cnt in list(total_counts.most_common()):
if(cnt > 100):
pos_neg_ratio = positive_counts[term] / float(negative_counts[term]+1)
pos_neg_ratios[term] = pos_neg_ratio

for word,ratio in pos_neg_ratios.most_common():
if(ratio > 1):
pos_neg_ratios[word] = np.log(ratio)
else:
pos_neg_ratios[word] = -np.log((1 / (ratio+0.01)))

``````
``````

In [11]:

# words most frequently seen in a review with a "POSITIVE" label
pos_neg_ratios.most_common()[:15]

``````
``````

Out[11]:

[('edie', 4.6913478822291435),
('paulie', 4.0775374439057197),
('felix', 3.1527360223636558),
('polanski', 2.8233610476132043),
('matthau', 2.8067217286092401),
('victoria', 2.6810215287142909),
('mildred', 2.6026896854443837),
('gandhi', 2.5389738710582761),
('flawless', 2.451005098112319),
('superbly', 2.2600254785752498),
('perfection', 2.1594842493533721),
('astaire', 2.1400661634962708),
('captures', 2.0386195471595809),
('voight', 2.0301704926730531),
('wonderfully', 2.0218960560332353)]

``````
``````

In [12]:

# words most frequently seen in a review with a "NEGATIVE" label
list(reversed(pos_neg_ratios.most_common()))[0:30]

``````
``````

Out[12]:

[('boll', -4.0778152602708904),
('uwe', -3.9218753018711578),
('seagal', -3.3202501058581921),
('unwatchable', -3.0269848170580955),
('stinker', -2.9876839403711624),
('mst', -2.7753833211707968),
('incoherent', -2.7641396677532537),
('unfunny', -2.5545257844967644),
('waste', -2.4907515123361046),
('blah', -2.4475792789485005),
('horrid', -2.3715779644809971),
('pointless', -2.3451073877136341),
('atrocious', -2.3187369339642556),
('redeeming', -2.2667790015910296),
('prom', -2.2601040980178784),
('drivel', -2.2476029585766928),
('lousy', -2.2118080125207054),
('worst', -2.1930856334332267),
('laughable', -2.172468615469592),
('awful', -2.1385076866397488),
('poorly', -2.1326133844207011),
('wasting', -2.1178155545614512),
('remotely', -2.111046881095167),
('existent', -2.0024805005437076),
('boredom', -1.9241486572738005),
('miserably', -1.9216610938019989),
('sucks', -1.9166645809588516),
('uninspired', -1.9131499212248517),
('lame', -1.9117232884159072),
('insult', -1.9085323769376259)]

``````

# Transforming Text into Numbers

``````

In [13]:

from IPython.display import Image

review = "This was a horrible, terrible movie."

Image(filename='sentiment_network.png')

``````
``````

Out[13]:

``````
``````

In [14]:

review = "The movie was excellent"

Image(filename='sentiment_network_pos.png')

``````
``````

Out[14]:

``````

# Project 2: Creating the Input/Output Data

``````

In [15]:

vocab = set(total_counts.keys())
vocab_size = len(vocab)
print(vocab_size)

``````
``````

74074

``````
``````

In [16]:

list(vocab)[:15]

``````
``````

Out[16]:

['',
'puyn',
'yalu',
'metaphysically',
'noughts',
'vending',
'ditch',
'delectably',
'luckier',
'chace',
'selling',
'california',
'sacker',
'vandalised',
'euro']

``````
``````

In [17]:

import numpy as np

layer_0 = np.zeros((1,vocab_size))
layer_0

``````
``````

Out[17]:

array([[ 0.,  0.,  0., ...,  0.,  0.,  0.]])

``````
``````

In [18]:

from IPython.display import Image
Image(filename='sentiment_network.png')

``````
``````

Out[18]:

``````
``````

In [19]:

word2index = {}

for i,word in enumerate(vocab):
word2index[word] = i

word2index_sample = {k: word2index[k] for k in list(word2index.keys())[:15]}

word2index_sample

``````
``````

Out[19]:

{'': 0,
'california': 11,
'chace': 9,
'delectably': 7,
'ditch': 6,
'euro': 14,
'luckier': 8,
'metaphysically': 3,
'noughts': 4,
'puyn': 1,
'sacker': 12,
'selling': 10,
'vandalised': 13,
'vending': 5,
'yalu': 2}

``````
``````

In [20]:

def update_input_layer(review):

global layer_0

# clear out previous state, reset the layer to be all 0s
layer_0 *= 0
for word in review.split(" "):
layer_0[0][word2index[word]] += 1

update_input_layer(reviews[0])

``````
``````

In [21]:

layer_0

``````
``````

Out[21]:

array([[ 18.,   0.,   0., ...,   0.,   0.,   0.]])

``````
``````

In [22]:

def get_target_for_label(label):
if(label == 'POSITIVE'):
return 1
else:
return 0

``````
``````

In [23]:

labels[0]

``````
``````

Out[23]:

'POSITIVE'

``````
``````

In [24]:

get_target_for_label(labels[0])

``````
``````

Out[24]:

1

``````
``````

In [25]:

labels[1]

``````
``````

Out[25]:

'NEGATIVE'

``````
``````

In [26]:

get_target_for_label(labels[1])

``````
``````

Out[26]:

0

``````

# Project 3: Building a Neural Network

• 3 layer neural network
• no non-linearity in hidden layer
• use our functions to create the training data
• create a "pre_process_data" function to create vocabulary for our training data generating functions
• modify "train" to train over the entire corpus

### Where to Get Help if You Need it

``````

In [27]:

import time
import sys
import numpy as np

# Let's tweak our network from before to model these phenomena
class SentimentNetwork:
def __init__(self, reviews,labels,hidden_nodes = 10, learning_rate = 0.1):

# set our random number generator
np.random.seed(1)

self.pre_process_data(reviews, labels)

self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

def pre_process_data(self, reviews, labels):

review_vocab = set()
for review in reviews:
for word in review.split(" "):
self.review_vocab = list(review_vocab)

label_vocab = set()
for label in labels:

self.label_vocab = list(label_vocab)

self.review_vocab_size = len(self.review_vocab)
self.label_vocab_size = len(self.label_vocab)

self.word2index = {}
for i, word in enumerate(self.review_vocab):
self.word2index[word] = i

self.label2index = {}
for i, label in enumerate(self.label_vocab):
self.label2index[label] = i

def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
# Set number of nodes in input, hidden and output layers.
self.input_nodes = input_nodes
self.hidden_nodes = hidden_nodes
self.output_nodes = output_nodes

# Initialize weights
self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))

self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,
(self.hidden_nodes, self.output_nodes))

self.learning_rate = learning_rate

self.layer_0 = np.zeros((1,input_nodes))

def update_input_layer(self,review):

# clear out previous state, reset the layer to be all 0s
self.layer_0 *= 0
for word in review.split(" "):
if(word in self.word2index.keys()):
self.layer_0[0][self.word2index[word]] += 1

def get_target_for_label(self,label):
if(label == 'POSITIVE'):
return 1
else:
return 0

def sigmoid(self,x):
return 1 / (1 + np.exp(-x))

def sigmoid_output_2_derivative(self,output):
return output * (1 - output)

def train(self, training_reviews, training_labels):

assert(len(training_reviews) == len(training_labels))

correct_so_far = 0

start = time.time()

for i in range(len(training_reviews)):

review = training_reviews[i]
label = training_labels[i]

#### Implement the forward pass here ####
### Forward pass ###

# Input Layer
self.update_input_layer(review)

# Hidden layer
layer_1 = self.layer_0.dot(self.weights_0_1)

# Output layer
layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))

#### Implement the backward pass here ####
### Backward pass ###

# TODO: Output error
layer_2_error = layer_2 - self.get_target_for_label(label) # Output layer error is the difference between desired target and actual output.
layer_2_delta = layer_2_error * self.sigmoid_output_2_derivative(layer_2)

# TODO: Backpropagated error
layer_1_error = layer_2_delta.dot(self.weights_1_2.T) # errors propagated to the hidden layer
layer_1_delta = layer_1_error # hidden layer gradients - no nonlinearity so it's the same as the error

# TODO: Update the weights
self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step

if(np.abs(layer_2_error) < 0.5):
correct_so_far += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(training_reviews)))[:4] + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] + " #Correct:" + str(correct_so_far) + " #Trained:" + str(i+1) + " Training Accuracy:" + str(correct_so_far * 100 / float(i+1))[:4] + "%")
if(i % 2500 == 0):
print("")

def test(self, testing_reviews, testing_labels):

correct = 0

start = time.time()

for i in range(len(testing_reviews)):
pred = self.run(testing_reviews[i])
if(pred == testing_labels[i]):
correct += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(testing_reviews)))[:4] \
+ "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] \
+ "% #Correct:" + str(correct) + " #Tested:" + str(i+1) + " Testing Accuracy:" + str(correct * 100 / float(i+1))[:4] + "%")

def run(self, review):

# Input Layer
self.update_input_layer(review.lower())

# Hidden layer
layer_1 = self.layer_0.dot(self.weights_0_1)

# Output layer
layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))

if(layer_2[0] > 0.5):
return "POSITIVE"
else:
return "NEGATIVE"

``````
``````

In [29]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)

``````
``````

In [61]:

# evaluate our model before training (just to show how horrible it is)
mlp.test(reviews[-1000:],labels[-1000:])

``````
``````

Progress:99.9% Speed(reviews/sec):587.5% #Correct:500 #Tested:1000 Testing Accuracy:50.0%

``````
``````

In [62]:

# train the network
mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:0.0% Speed(reviews/sec):0.0 #Correct:0 #Trained:1 Training Accuracy:0.0%
Progress:10.4% Speed(reviews/sec):89.58 #Correct:1250 #Trained:2501 Training Accuracy:49.9%
Progress:20.8% Speed(reviews/sec):95.03 #Correct:2500 #Trained:5001 Training Accuracy:49.9%
Progress:27.4% Speed(reviews/sec):95.46 #Correct:3295 #Trained:6592 Training Accuracy:49.9%

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
1 # train the network
----> 2 mlp.train(reviews[:-1000],labels[:-1000])

<ipython-input-59-6334c4ec4642> in train(self, training_reviews, training_labels)
117             # TODO: Update the weights
118             self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
--> 119             self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step
120
121             if(np.abs(layer_2_error) < 0.5):

KeyboardInterrupt:

``````
``````

In [63]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.01)

``````
``````

In [64]:

# train the network
mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:0.0% Speed(reviews/sec):0.0 #Correct:0 #Trained:1 Training Accuracy:0.0%
Progress:10.4% Speed(reviews/sec):96.39 #Correct:1247 #Trained:2501 Training Accuracy:49.8%
Progress:20.8% Speed(reviews/sec):99.31 #Correct:2497 #Trained:5001 Training Accuracy:49.9%
Progress:22.8% Speed(reviews/sec):99.02 #Correct:2735 #Trained:5476 Training Accuracy:49.9%

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
1 # train the network
----> 2 mlp.train(reviews[:-1000],labels[:-1000])

<ipython-input-59-6334c4ec4642> in train(self, training_reviews, training_labels)
117             # TODO: Update the weights
118             self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
--> 119             self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step
120
121             if(np.abs(layer_2_error) < 0.5):

KeyboardInterrupt:

``````
``````

In [65]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.001)

``````
``````

In [66]:

# train the network
mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:0.0% Speed(reviews/sec):0.0 #Correct:0 #Trained:1 Training Accuracy:0.0%
Progress:10.4% Speed(reviews/sec):98.77 #Correct:1267 #Trained:2501 Training Accuracy:50.6%
Progress:20.8% Speed(reviews/sec):98.79 #Correct:2640 #Trained:5001 Training Accuracy:52.7%
Progress:31.2% Speed(reviews/sec):98.58 #Correct:4109 #Trained:7501 Training Accuracy:54.7%
Progress:41.6% Speed(reviews/sec):93.78 #Correct:5638 #Trained:10001 Training Accuracy:56.3%
Progress:52.0% Speed(reviews/sec):91.76 #Correct:7246 #Trained:12501 Training Accuracy:57.9%
Progress:62.5% Speed(reviews/sec):92.42 #Correct:8841 #Trained:15001 Training Accuracy:58.9%
Progress:69.4% Speed(reviews/sec):92.58 #Correct:9934 #Trained:16668 Training Accuracy:59.5%

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
1 # train the network
----> 2 mlp.train(reviews[:-1000],labels[:-1000])

<ipython-input-59-6334c4ec4642> in train(self, training_reviews, training_labels)
117             # TODO: Update the weights
118             self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
--> 119             self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step
120
121             if(np.abs(layer_2_error) < 0.5):

KeyboardInterrupt:

``````

# Understanding Neural Noise

``````

In [28]:

from IPython.display import Image
Image(filename='sentiment_network.png')

``````
``````

Out[28]:

``````
``````

In [29]:

def update_input_layer(review):

global layer_0

# clear out previous state, reset the layer to be all 0s
layer_0 *= 0
for word in review.split(" "):
layer_0[0][word2index[word]] += 1

update_input_layer(reviews[0])

``````
``````

In [30]:

layer_0

``````
``````

Out[30]:

array([[ 18.,   0.,   0., ...,   0.,   0.,   0.]])

``````
``````

In [31]:

review_counter = Counter()

``````
``````

In [32]:

for word in reviews[0].split(" "):
review_counter[word] += 1

``````
``````

In [33]:

review_counter.most_common()[:15]

``````
``````

Out[33]:

[('.', 27),
('', 18),
('the', 9),
('to', 6),
('high', 5),
('i', 5),
('bromwell', 4),
('is', 4),
('a', 4),
('teachers', 4),
('that', 4),
('of', 4),
('it', 2),
('at', 2),
('as', 2)]

``````

# Project 4: Reducing Noise in our Input Data

``````

In [34]:

import time
import sys
import numpy as np

# Let's tweak our network from before to model these phenomena
class SentimentNetwork:
def __init__(self, reviews,labels,hidden_nodes = 10, learning_rate = 0.1):

# set our random number generator
np.random.seed(1)

self.pre_process_data(reviews, labels)

self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

def pre_process_data(self, reviews, labels):

review_vocab = set()
for review in reviews:
for word in review.split(" "):
self.review_vocab = list(review_vocab)

label_vocab = set()
for label in labels:

self.label_vocab = list(label_vocab)

self.review_vocab_size = len(self.review_vocab)
self.label_vocab_size = len(self.label_vocab)

self.word2index = {}
for i, word in enumerate(self.review_vocab):
self.word2index[word] = i

self.label2index = {}
for i, label in enumerate(self.label_vocab):
self.label2index[label] = i

def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
# Set number of nodes in input, hidden and output layers.
self.input_nodes = input_nodes
self.hidden_nodes = hidden_nodes
self.output_nodes = output_nodes

# Initialize weights
self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))

self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,
(self.hidden_nodes, self.output_nodes))

self.learning_rate = learning_rate

self.layer_0 = np.zeros((1,input_nodes))

def update_input_layer(self,review):

# clear out previous state, reset the layer to be all 0s
self.layer_0 *= 0
for word in review.split(" "):
if(word in self.word2index.keys()):
self.layer_0[0][self.word2index[word]] = 1

def get_target_for_label(self,label):
if(label == 'POSITIVE'):
return 1
else:
return 0

def sigmoid(self,x):
return 1 / (1 + np.exp(-x))

def sigmoid_output_2_derivative(self,output):
return output * (1 - output)

def train(self, training_reviews, training_labels):

assert(len(training_reviews) == len(training_labels))

correct_so_far = 0

start = time.time()

for i in range(len(training_reviews)):

review = training_reviews[i]
label = training_labels[i]

#### Implement the forward pass here ####
### Forward pass ###

# Input Layer
self.update_input_layer(review)

# Hidden layer
layer_1 = self.layer_0.dot(self.weights_0_1)

# Output layer
layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))

#### Implement the backward pass here ####
### Backward pass ###

# TODO: Output error
layer_2_error = layer_2 - self.get_target_for_label(label) # Output layer error is the difference between desired target and actual output.
layer_2_delta = layer_2_error * self.sigmoid_output_2_derivative(layer_2)

# TODO: Backpropagated error
layer_1_error = layer_2_delta.dot(self.weights_1_2.T) # errors propagated to the hidden layer
layer_1_delta = layer_1_error # hidden layer gradients - no nonlinearity so it's the same as the error

# TODO: Update the weights
self.weights_1_2 -= layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step
self.weights_0_1 -= self.layer_0.T.dot(layer_1_delta) * self.learning_rate # update input-to-hidden weights with gradient descent step

if(np.abs(layer_2_error) < 0.5):
correct_so_far += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(training_reviews)))[:4] + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] + " #Correct:" + str(correct_so_far) + " #Trained:" + str(i+1) + " Training Accuracy:" + str(correct_so_far * 100 / float(i+1))[:4] + "%")
if(i % 2500 == 0):
print("")

def test(self, testing_reviews, testing_labels):

correct = 0

start = time.time()

for i in range(len(testing_reviews)):
pred = self.run(testing_reviews[i])
if(pred == testing_labels[i]):
correct += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(testing_reviews)))[:4] \
+ "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] \
+ "% #Correct:" + str(correct) + " #Tested:" + str(i+1) + " Testing Accuracy:" + str(correct * 100 / float(i+1))[:4] + "%")

def run(self, review):

# Input Layer
self.update_input_layer(review.lower())

# Hidden layer
layer_1 = self.layer_0.dot(self.weights_0_1)

# Output layer
layer_2 = self.sigmoid(layer_1.dot(self.weights_1_2))

if(layer_2[0] > 0.5):
return "POSITIVE"
else:
return "NEGATIVE"

``````
``````

In [37]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)

``````
``````

In [84]:

mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:0.0% Speed(reviews/sec):0.0 #Correct:0 #Trained:1 Training Accuracy:0.0%
Progress:10.4% Speed(reviews/sec):91.50 #Correct:1795 #Trained:2501 Training Accuracy:71.7%
Progress:20.8% Speed(reviews/sec):95.25 #Correct:3811 #Trained:5001 Training Accuracy:76.2%
Progress:31.2% Speed(reviews/sec):93.74 #Correct:5898 #Trained:7501 Training Accuracy:78.6%
Progress:41.6% Speed(reviews/sec):93.69 #Correct:8042 #Trained:10001 Training Accuracy:80.4%
Progress:52.0% Speed(reviews/sec):95.27 #Correct:10186 #Trained:12501 Training Accuracy:81.4%
Progress:62.5% Speed(reviews/sec):98.19 #Correct:12317 #Trained:15001 Training Accuracy:82.1%
Progress:72.9% Speed(reviews/sec):98.56 #Correct:14440 #Trained:17501 Training Accuracy:82.5%
Progress:83.3% Speed(reviews/sec):99.74 #Correct:16613 #Trained:20001 Training Accuracy:83.0%
Progress:93.7% Speed(reviews/sec):100.7 #Correct:18794 #Trained:22501 Training Accuracy:83.5%
Progress:99.9% Speed(reviews/sec):101.9 #Correct:20115 #Trained:24000 Training Accuracy:83.8%

``````
``````

In [85]:

# evaluate our model before training (just to show how horrible it is)
mlp.test(reviews[-1000:],labels[-1000:])

``````
``````

Progress:99.9% Speed(reviews/sec):832.7% #Correct:851 #Tested:1000 Testing Accuracy:85.1%

``````

# Analyzing Inefficiencies in our Network

``````

In [35]:

Image(filename='sentiment_network_sparse.png')

``````
``````

Out[35]:

``````
``````

In [36]:

layer_0 = np.zeros(10)

``````
``````

In [37]:

layer_0

``````
``````

Out[37]:

array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

``````
``````

In [38]:

layer_0[4] = 1
layer_0[9] = 1

``````
``````

In [39]:

layer_0

``````
``````

Out[39]:

array([ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  1.])

``````
``````

In [40]:

weights_0_1 = np.random.randn(10,5)

``````
``````

In [41]:

layer_0.dot(weights_0_1)

``````
``````

Out[41]:

array([-0.16464196, -0.23818699,  1.75822623, -0.08466335, -1.7099541 ])

``````
``````

In [42]:

indices = [4,9]

``````
``````

In [43]:

layer_1 = np.zeros(5)

``````
``````

In [44]:

for index in indices:
layer_1 += (weights_0_1[index])

``````
``````

In [45]:

layer_1

``````
``````

Out[45]:

array([-0.16464196, -0.23818699,  1.75822623, -0.08466335, -1.7099541 ])

``````
``````

In [46]:

Image(filename='sentiment_network_sparse_2.png')

``````
``````

Out[46]:

``````

# Project 5: Making our Network More Efficient

``````

In [47]:

import time
import sys

# Let's tweak our network from before to model these phenomena
class SentimentNetwork:
def __init__(self, reviews,labels,hidden_nodes = 10, learning_rate = 0.1):

np.random.seed(1)

self.pre_process_data(reviews)

self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

def pre_process_data(self,reviews):

review_vocab = set()
for review in reviews:
for word in review.split(" "):
self.review_vocab = list(review_vocab)

label_vocab = set()
for label in labels:

self.label_vocab = list(label_vocab)

self.review_vocab_size = len(self.review_vocab)
self.label_vocab_size = len(self.label_vocab)

self.word2index = {}
for i, word in enumerate(self.review_vocab):
self.word2index[word] = i

self.label2index = {}
for i, label in enumerate(self.label_vocab):
self.label2index[label] = i

def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
# Set number of nodes in input, hidden and output layers.
self.input_nodes = input_nodes
self.hidden_nodes = hidden_nodes
self.output_nodes = output_nodes

# Initialize weights
self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))

self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,
(self.hidden_nodes, self.output_nodes))

self.learning_rate = learning_rate

self.layer_0 = np.zeros((1,input_nodes))
self.layer_1 = np.zeros((1,hidden_nodes))

def sigmoid(self,x):
return 1 / (1 + np.exp(-x))

def sigmoid_output_2_derivative(self,output):
return output * (1 - output)

def update_input_layer(self,review):

# clear out previous state, reset the layer to be all 0s
self.layer_0 *= 0
for word in review.split(" "):
self.layer_0[0][self.word2index[word]] = 1

def get_target_for_label(self,label):
if(label == 'POSITIVE'):
return 1
else:
return 0

def train(self, training_reviews_raw, training_labels):

training_reviews = list()
for review in training_reviews_raw:
indices = set()
for word in review.split(" "):
if(word in self.word2index.keys()):
training_reviews.append(list(indices))

assert(len(training_reviews) == len(training_labels))

correct_so_far = 0

start = time.time()

for i in range(len(training_reviews)):

review = training_reviews[i]
label = training_labels[i]

#### Implement the forward pass here ####
### Forward pass ###

# Input Layer

# Hidden layer
#             layer_1 = self.layer_0.dot(self.weights_0_1)
self.layer_1 *= 0
for index in review:
self.layer_1 += self.weights_0_1[index]

# Output layer
layer_2 = self.sigmoid(self.layer_1.dot(self.weights_1_2))

#### Implement the backward pass here ####
### Backward pass ###

# Output error
layer_2_error = layer_2 - self.get_target_for_label(label) # Output layer error is the difference between desired target and actual output.
layer_2_delta = layer_2_error * self.sigmoid_output_2_derivative(layer_2)

# Backpropagated error
layer_1_error = layer_2_delta.dot(self.weights_1_2.T) # errors propagated to the hidden layer
layer_1_delta = layer_1_error # hidden layer gradients - no nonlinearity so it's the same as the error

# Update the weights
self.weights_1_2 -= self.layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step

for index in review:
self.weights_0_1[index] -= layer_1_delta[0] * self.learning_rate # update input-to-hidden weights with gradient descent step

if(np.abs(layer_2_error) < 0.5):
correct_so_far += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(training_reviews)))[:4] + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] + " #Correct:" + str(correct_so_far) + " #Trained:" + str(i+1) + " Training Accuracy:" + str(correct_so_far * 100 / float(i+1))[:4] + "%")

def test(self, testing_reviews, testing_labels):

correct = 0

start = time.time()

for i in range(len(testing_reviews)):
pred = self.run(testing_reviews[i])
if(pred == testing_labels[i]):
correct += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(testing_reviews)))[:4] \
+ "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] \
+ "% #Correct:" + str(correct) + " #Tested:" + str(i+1) + " Testing Accuracy:" + str(correct * 100 / float(i+1))[:4] + "%")

def run(self, review):

# Input Layer

# Hidden layer
self.layer_1 *= 0
unique_indices = set()
for word in review.lower().split(" "):
if word in self.word2index.keys():
for index in unique_indices:
self.layer_1 += self.weights_0_1[index]

# Output layer
layer_2 = self.sigmoid(self.layer_1.dot(self.weights_1_2))

if(layer_2[0] > 0.5):
return "POSITIVE"
else:
return "NEGATIVE"

``````
``````

In [51]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)

``````
``````

In [111]:

mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

In [109]:

# evaluate our model before training (just to show how horrible it is)
mlp.test(reviews[-1000:],labels[-1000:])

``````
``````

Progress:99.9% Speed(reviews/sec):1581.% #Correct:857 #Tested:1000 Testing Accuracy:85.7%

``````

# Further Noise Reduction

``````

In [48]:

Image(filename='sentiment_network_sparse_2.png')

``````
``````

Out[48]:

``````
``````

In [49]:

# words most frequently seen in a review with a "POSITIVE" label
pos_neg_ratios.most_common()[:15]

``````
``````

Out[49]:

[('edie', 4.6913478822291435),
('paulie', 4.0775374439057197),
('felix', 3.1527360223636558),
('polanski', 2.8233610476132043),
('matthau', 2.8067217286092401),
('victoria', 2.6810215287142909),
('mildred', 2.6026896854443837),
('gandhi', 2.5389738710582761),
('flawless', 2.451005098112319),
('superbly', 2.2600254785752498),
('perfection', 2.1594842493533721),
('astaire', 2.1400661634962708),
('captures', 2.0386195471595809),
('voight', 2.0301704926730531),
('wonderfully', 2.0218960560332353)]

``````
``````

In [50]:

# words most frequently seen in a review with a "NEGATIVE" label
list(reversed(pos_neg_ratios.most_common()))[0:30]

``````
``````

Out[50]:

[('boll', -4.0778152602708904),
('uwe', -3.9218753018711578),
('seagal', -3.3202501058581921),
('unwatchable', -3.0269848170580955),
('stinker', -2.9876839403711624),
('mst', -2.7753833211707968),
('incoherent', -2.7641396677532537),
('unfunny', -2.5545257844967644),
('waste', -2.4907515123361046),
('blah', -2.4475792789485005),
('horrid', -2.3715779644809971),
('pointless', -2.3451073877136341),
('atrocious', -2.3187369339642556),
('redeeming', -2.2667790015910296),
('prom', -2.2601040980178784),
('drivel', -2.2476029585766928),
('lousy', -2.2118080125207054),
('worst', -2.1930856334332267),
('laughable', -2.172468615469592),
('awful', -2.1385076866397488),
('poorly', -2.1326133844207011),
('wasting', -2.1178155545614512),
('remotely', -2.111046881095167),
('existent', -2.0024805005437076),
('boredom', -1.9241486572738005),
('miserably', -1.9216610938019989),
('sucks', -1.9166645809588516),
('uninspired', -1.9131499212248517),
('lame', -1.9117232884159072),
('insult', -1.9085323769376259)]

``````
``````

In [51]:

from bokeh.models import ColumnDataSource, LabelSet
from bokeh.plotting import figure, show, output_file
from bokeh.io import output_notebook
output_notebook()

``````
``````

var element = \$('#bf422d1a-41db-447e-b39f-78c786798941');

(function(global) {
function now() {
return new Date();
}

var force = true;

if (typeof (window._bokeh_onload_callbacks) === "undefined" || force === true) {
}

if (typeof (window._bokeh_timeout) === "undefined" || force === true) {
window._bokeh_timeout = Date.now() + 5000;
}

"<div style='background-color: #fdd'>\n"+
"<p>\n"+
"may be due to a slow or bad network connection. Possible fixes:\n"+
"</p>\n"+
"<ul>\n"+
"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\n"+
"<li>use INLINE resources instead, as so:</li>\n"+
"</ul>\n"+
"<code>\n"+
"from bokeh.resources import INLINE\n"+
"output_notebook(resources=INLINE)\n"+
"</code>\n"+
"</div>"}};

if (window.Bokeh !== undefined) {
} else if (Date.now() < window._bokeh_timeout) {
}
}

function run_callbacks() {
console.info("Bokeh: all callbacks have finished");
}

console.log("Bokeh: BokehJS is being loaded, scheduling callback at", now());
return null;
}
if (js_urls == null || js_urls.length === 0) {
run_callbacks();
return null;
}
for (var i = 0; i < js_urls.length; i++) {
var url = js_urls[i];
var s = document.createElement('script');
s.src = url;
s.async = false;
run_callbacks()
}
};
s.onerror = function() {
console.warn("failed to load library " + url);
};
console.log("Bokeh: injecting script tag for BokehJS library: ", url);
}
};var element = document.getElementById("e634b10f-1526-44df-b71e-c7dee10614b7");
if (element == null) {
console.log("Bokeh: ERROR: autoload.js configured with elementid 'e634b10f-1526-44df-b71e-c7dee10614b7' but no matching script tag was found. ")
return false;
}

var js_urls = ["https://cdn.pydata.org/bokeh/release/bokeh-0.12.4.min.js", "https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.4.min.js"];

var inline_js = [
function(Bokeh) {
Bokeh.set_log_level("info");
},

function(Bokeh) {

},
function(Bokeh) {
console.log("Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-0.12.4.min.css");
Bokeh.embed.inject_css("https://cdn.pydata.org/bokeh/release/bokeh-0.12.4.min.css");
console.log("Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.4.min.css");
Bokeh.embed.inject_css("https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.4.min.css");
}
];

function run_inline_js() {

if ((window.Bokeh !== undefined) || (force === true)) {
for (var i = 0; i < inline_js.length; i++) {
inline_js[i](window.Bokeh);
}if (force === true) {
}} else if (Date.now() < window._bokeh_timeout) {
setTimeout(run_inline_js, 100);
console.log("Bokeh: BokehJS failed to load within specified timeout.");
} else if (force !== true) {
var cell = \$(document.getElementById("e634b10f-1526-44df-b71e-c7dee10614b7")).parents('.cell').data().cell;
}

}

console.log("Bokeh: BokehJS loaded, going straight to plotting");
run_inline_js();
} else {
console.log("Bokeh: BokehJS plotting callback run at", now());
run_inline_js();
});
}
}(this));

``````
``````

In [52]:

hist, edges = np.histogram(list(map(lambda x:x[1],pos_neg_ratios.most_common())), density=True, bins=100, normed=True)

p = figure(tools="pan,wheel_zoom,reset,save",
toolbar_location="above",
title="Word Positive/Negative Affinity Distribution")
show(p)

``````
``````

(function(global) {
function now() {
return new Date();
}

var force = false;

if (typeof (window._bokeh_onload_callbacks) === "undefined" || force === true) {
}

if (typeof (window._bokeh_timeout) === "undefined" || force === true) {
window._bokeh_timeout = Date.now() + 0;
}

"<div style='background-color: #fdd'>\n"+
"<p>\n"+
"may be due to a slow or bad network connection. Possible fixes:\n"+
"</p>\n"+
"<ul>\n"+
"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\n"+
"<li>use INLINE resources instead, as so:</li>\n"+
"</ul>\n"+
"<code>\n"+
"from bokeh.resources import INLINE\n"+
"output_notebook(resources=INLINE)\n"+
"</code>\n"+
"</div>"}};

if (window.Bokeh !== undefined) {
} else if (Date.now() < window._bokeh_timeout) {
}
}

function run_callbacks() {
console.info("Bokeh: all callbacks have finished");
}

console.log("Bokeh: BokehJS is being loaded, scheduling callback at", now());
return null;
}
if (js_urls == null || js_urls.length === 0) {
run_callbacks();
return null;
}
for (var i = 0; i < js_urls.length; i++) {
var url = js_urls[i];
var s = document.createElement('script');
s.src = url;
s.async = false;
run_callbacks()
}
};
s.onerror = function() {
console.warn("failed to load library " + url);
};
console.log("Bokeh: injecting script tag for BokehJS library: ", url);
}
};var element = document.getElementById("c70e5fb8-ff6e-4444-b077-3f59ef13f117");
if (element == null) {
console.log("Bokeh: ERROR: autoload.js configured with elementid 'c70e5fb8-ff6e-4444-b077-3f59ef13f117' but no matching script tag was found. ")
return false;
}

var js_urls = [];

var inline_js = [
function(Bokeh) {
(function() {
var fn = function() {

Bokeh.embed.embed_items(docs_json, render_items);
};
})();
},
function(Bokeh) {
}
];

function run_inline_js() {

if ((window.Bokeh !== undefined) || (force === true)) {
for (var i = 0; i < inline_js.length; i++) {
inline_js[i](window.Bokeh);
}if (force === true) {
}} else if (Date.now() < window._bokeh_timeout) {
setTimeout(run_inline_js, 100);
console.log("Bokeh: BokehJS failed to load within specified timeout.");
} else if (force !== true) {
var cell = \$(document.getElementById("c70e5fb8-ff6e-4444-b077-3f59ef13f117")).parents('.cell').data().cell;
}

}

console.log("Bokeh: BokehJS loaded, going straight to plotting");
run_inline_js();
} else {
console.log("Bokeh: BokehJS plotting callback run at", now());
run_inline_js();
});
}
}(this));

``````
``````

In [53]:

frequency_frequency = Counter()

for word, cnt in total_counts.most_common():
frequency_frequency[cnt] += 1

``````
``````

In [54]:

hist, edges = np.histogram(list(map(lambda x:x[1],frequency_frequency.most_common())), density=True, bins=100, normed=True)

p = figure(tools="pan,wheel_zoom,reset,save",
toolbar_location="above",
title="The frequency distribution of the words in our corpus")
show(p)

``````
``````

(function(global) {
function now() {
return new Date();
}

var force = false;

if (typeof (window._bokeh_onload_callbacks) === "undefined" || force === true) {
}

if (typeof (window._bokeh_timeout) === "undefined" || force === true) {
window._bokeh_timeout = Date.now() + 0;
}

"<div style='background-color: #fdd'>\n"+
"<p>\n"+
"may be due to a slow or bad network connection. Possible fixes:\n"+
"</p>\n"+
"<ul>\n"+
"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\n"+
"<li>use INLINE resources instead, as so:</li>\n"+
"</ul>\n"+
"<code>\n"+
"from bokeh.resources import INLINE\n"+
"output_notebook(resources=INLINE)\n"+
"</code>\n"+
"</div>"}};

if (window.Bokeh !== undefined) {
} else if (Date.now() < window._bokeh_timeout) {
}
}

function run_callbacks() {
console.info("Bokeh: all callbacks have finished");
}

console.log("Bokeh: BokehJS is being loaded, scheduling callback at", now());
return null;
}
if (js_urls == null || js_urls.length === 0) {
run_callbacks();
return null;
}
for (var i = 0; i < js_urls.length; i++) {
var url = js_urls[i];
var s = document.createElement('script');
s.src = url;
s.async = false;
run_callbacks()
}
};
s.onerror = function() {
console.warn("failed to load library " + url);
};
console.log("Bokeh: injecting script tag for BokehJS library: ", url);
}
};var element = document.getElementById("7dc27391-d4a8-41a4-a351-2b4026c8c79d");
if (element == null) {
console.log("Bokeh: ERROR: autoload.js configured with elementid '7dc27391-d4a8-41a4-a351-2b4026c8c79d' but no matching script tag was found. ")
return false;
}

var js_urls = [];

var inline_js = [
function(Bokeh) {
(function() {
var fn = function() {

Bokeh.embed.embed_items(docs_json, render_items);
};
})();
},
function(Bokeh) {
}
];

function run_inline_js() {

if ((window.Bokeh !== undefined) || (force === true)) {
for (var i = 0; i < inline_js.length; i++) {
inline_js[i](window.Bokeh);
}if (force === true) {
}} else if (Date.now() < window._bokeh_timeout) {
setTimeout(run_inline_js, 100);
console.log("Bokeh: BokehJS failed to load within specified timeout.");
} else if (force !== true) {
var cell = \$(document.getElementById("7dc27391-d4a8-41a4-a351-2b4026c8c79d")).parents('.cell').data().cell;
}

}

console.log("Bokeh: BokehJS loaded, going straight to plotting");
run_inline_js();
} else {
console.log("Bokeh: BokehJS plotting callback run at", now());
run_inline_js();
});
}
}(this));

``````

# Project 6: Reducing Noise by Strategically Reducing Vocabulary

``````

In [55]:

import time
import sys
import numpy as np

# Let's tweak our network from before to model these phenomena
class SentimentNetwork:
def __init__(self, reviews,labels,min_count = 10,polarity_cutoff = 0.1,hidden_nodes = 10, learning_rate = 0.1):

np.random.seed(1)

self.pre_process_data(reviews, polarity_cutoff, min_count)

self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)

def pre_process_data(self,reviews, polarity_cutoff,min_count):

positive_counts = Counter()
negative_counts = Counter()
total_counts = Counter()

for i in range(len(reviews)):
if(labels[i] == 'POSITIVE'):
for word in reviews[i].split(" "):
positive_counts[word] += 1
total_counts[word] += 1
else:
for word in reviews[i].split(" "):
negative_counts[word] += 1
total_counts[word] += 1

pos_neg_ratios = Counter()

for term,cnt in list(total_counts.most_common()):
if(cnt >= 50):
pos_neg_ratio = positive_counts[term] / float(negative_counts[term]+1)
pos_neg_ratios[term] = pos_neg_ratio

for word,ratio in pos_neg_ratios.most_common():
if(ratio > 1):
pos_neg_ratios[word] = np.log(ratio)
else:
pos_neg_ratios[word] = -np.log((1 / (ratio + 0.01)))

review_vocab = set()
for review in reviews:
for word in review.split(" "):
if(total_counts[word] > min_count):
if(word in pos_neg_ratios.keys()):
if((pos_neg_ratios[word] >= polarity_cutoff) or (pos_neg_ratios[word] <= -polarity_cutoff)):
else:
self.review_vocab = list(review_vocab)

label_vocab = set()
for label in labels:

self.label_vocab = list(label_vocab)

self.review_vocab_size = len(self.review_vocab)
self.label_vocab_size = len(self.label_vocab)

self.word2index = {}
for i, word in enumerate(self.review_vocab):
self.word2index[word] = i

self.label2index = {}
for i, label in enumerate(self.label_vocab):
self.label2index[label] = i

def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
# Set number of nodes in input, hidden and output layers.
self.input_nodes = input_nodes
self.hidden_nodes = hidden_nodes
self.output_nodes = output_nodes

# Initialize weights
self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))

self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,
(self.hidden_nodes, self.output_nodes))

self.learning_rate = learning_rate

self.layer_0 = np.zeros((1,input_nodes))
self.layer_1 = np.zeros((1,hidden_nodes))

def sigmoid(self,x):
return 1 / (1 + np.exp(-x))

def sigmoid_output_2_derivative(self,output):
return output * (1 - output)

def update_input_layer(self,review):

# clear out previous state, reset the layer to be all 0s
self.layer_0 *= 0
for word in review.split(" "):
self.layer_0[0][self.word2index[word]] = 1

def get_target_for_label(self,label):
if(label == 'POSITIVE'):
return 1
else:
return 0

def train(self, training_reviews_raw, training_labels):

training_reviews = list()
for review in training_reviews_raw:
indices = set()
for word in review.split(" "):
if(word in self.word2index.keys()):
training_reviews.append(list(indices))

assert(len(training_reviews) == len(training_labels))

correct_so_far = 0

start = time.time()

for i in range(len(training_reviews)):

review = training_reviews[i]
label = training_labels[i]

#### Implement the forward pass here ####
### Forward pass ###

# Input Layer

# Hidden layer
#             layer_1 = self.layer_0.dot(self.weights_0_1)
self.layer_1 *= 0
for index in review:
self.layer_1 += self.weights_0_1[index]

# Output layer
layer_2 = self.sigmoid(self.layer_1.dot(self.weights_1_2))

#### Implement the backward pass here ####
### Backward pass ###

# Output error
layer_2_error = layer_2 - self.get_target_for_label(label) # Output layer error is the difference between desired target and actual output.
layer_2_delta = layer_2_error * self.sigmoid_output_2_derivative(layer_2)

# Backpropagated error
layer_1_error = layer_2_delta.dot(self.weights_1_2.T) # errors propagated to the hidden layer
layer_1_delta = layer_1_error # hidden layer gradients - no nonlinearity so it's the same as the error

# Update the weights
self.weights_1_2 -= self.layer_1.T.dot(layer_2_delta) * self.learning_rate # update hidden-to-output weights with gradient descent step

for index in review:
self.weights_0_1[index] -= layer_1_delta[0] * self.learning_rate # update input-to-hidden weights with gradient descent step

if(layer_2 >= 0.5 and label == 'POSITIVE'):
correct_so_far += 1
if(layer_2 < 0.5 and label == 'NEGATIVE'):
correct_so_far += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(training_reviews)))[:4] + "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] + " #Correct:" + str(correct_so_far) + " #Trained:" + str(i+1) + " Training Accuracy:" + str(correct_so_far * 100 / float(i+1))[:4] + "%")

def test(self, testing_reviews, testing_labels):

correct = 0

start = time.time()

for i in range(len(testing_reviews)):
pred = self.run(testing_reviews[i])
if(pred == testing_labels[i]):
correct += 1

reviews_per_second = i / float(time.time() - start)

sys.stdout.write("\rProgress:" + str(100 * i/float(len(testing_reviews)))[:4] \
+ "% Speed(reviews/sec):" + str(reviews_per_second)[0:5] \
+ "% #Correct:" + str(correct) + " #Tested:" + str(i+1) + " Testing Accuracy:" + str(correct * 100 / float(i+1))[:4] + "%")

def run(self, review):

# Input Layer

# Hidden layer
self.layer_1 *= 0
unique_indices = set()
for word in review.lower().split(" "):
if word in self.word2index.keys():
for index in unique_indices:
self.layer_1 += self.weights_0_1[index]

# Output layer
layer_2 = self.sigmoid(self.layer_1.dot(self.weights_1_2))

if(layer_2[0] >= 0.5):
return "POSITIVE"
else:
return "NEGATIVE"

``````
``````

In [69]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000],min_count=20,polarity_cutoff=0.05,learning_rate=0.01)

``````
``````

In [70]:

mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:99.9% Speed(reviews/sec):1013. #Correct:20461 #Trained:24000 Training Accuracy:85.2%

``````
``````

In [71]:

mlp.test(reviews[-1000:],labels[-1000:])

``````
``````

Progress:99.9% Speed(reviews/sec):1354.% #Correct:859 #Tested:1000 Testing Accuracy:85.9%

``````
``````

In [75]:

mlp = SentimentNetwork(reviews[:-1000],labels[:-1000],min_count=20,polarity_cutoff=0.7,learning_rate=0.01)

``````
``````

In [76]:

mlp.train(reviews[:-1000],labels[:-1000])

``````
``````

Progress:99.9% Speed(reviews/sec):3509. #Correct:20712 #Trained:24000 Training Accuracy:86.3%

``````
``````

In [77]:

mlp.test(reviews[-1000:],labels[-1000:])

``````
``````

Progress:99.9% Speed(reviews/sec):3539.% #Correct:830 #Tested:1000 Testing Accuracy:83.0%

``````
``````

In [ ]:

``````