In [ ]:
# You will probably need to !pip install some of these
# !pip install scipy
# !pip install sklearn
# !pip install nltk
In [4]:
import pandas as pd
!pip install sklearn
!pip install scipy
!pip install nltk
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
import re #re: module for regular expressions
from nltk.stem.porter import PorterStemmer
pd.options.display.max_columns = 30
%matplotlib inline
In [5]:
texts = [
"Penny bought bright blue fishes.",
"Penny bought bright blue and orange fish.",
"The cat ate a fish at the store.",
"Penny went to the store. Penny ate a bug. Penny saw a fish.",
"It meowed once at the bug, it is still meowing at the bug and the fish",
"The cat is at the store. The cat is orange. The cat is meowing at the fish.",
"Penny is a fish"
]
When you process text, you have a nice long series of steps, but let's say you're interested in three things:
meow appear?meow appears vs. how many total words there are, or maybe you're seeing how often meow comes up to see whether it's important.
In [ ]:
Penny bought bright blue fishes
tokenized - penny bought bright blue fishes
In [13]:
"Penny bought bright blue fishes".split()
Out[13]:
The scikit-learn package does a ton of stuff, some of which includes the above. We're going to start by playing with the CountVectorizer.
In [14]:
from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer()
In [15]:
# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)
Let's take a look at what it found out!
In [17]:
X
Out[17]:
Okay, that looks like trash and garbage. What's a "sparse array"??????
In [18]:
X.toarray()
Out[18]:
If we put on our Computer Goggles we see that the first sentence has the first word 3 times, the second word 1 time, the third word 1 time, etc... But we can't read it, really. It would look nicer as a dataframe.
In [19]:
pd.DataFrame(X.toarray())
Out[19]:
What do all of those numbers mean????
In [20]:
# A fish is Penny
count_vectorizer.get_feature_names()
Out[20]:
In [21]:
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
Out[21]:
So sentence #4 has "at" once, and the first sentence has "bought" once, and the last sentence has "the" three times. But hey, those are garbage words! They're cluttering up our dataframe! We need to add stopwords!
In [22]:
# We'll make a new vectorizer
count_vectorizer = CountVectorizer(stop_words='english')
# .fit_transfer TOKENIZES and COUNTS
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
In [23]:
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
Out[23]:
I still see meowed and meowing and fish and fishes - they seem the same, so let's lemmatize/stem them.
You can specify a preprocessor or a tokenizer when you're creating your CountVectorizer to do custom stuff on your words. Maybe we want to get rid of punctuation, lowercase things and split them on spaces (this is basically the default). preprocessor is supposed to return a string, so it's a little easier to work with.
In [24]:
# This is what our normal tokenizer looks like
def boring_tokenizer(str_input):
words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
return words
count_vectorizer = CountVectorizer(stop_words='english', tokenizer=boring_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
We're going to use one that features a STEMMER - something that strips the endings off of words (or tries to, at least). This one is from nltk.
In [25]:
from nltk.stem.porter import PorterStemmer #it doesn't know what words are, it just chop off the ends from the words
porter_stemmer = PorterStemmer()
#
print(porter_stemmer.stem('fishes'))
print(porter_stemmer.stem('meowed'))
print(porter_stemmer.stem('oranges'))
print(porter_stemmer.stem('meowing'))
print(porter_stemmer.stem('organge'))
In [26]:
porter_stemmer = PorterStemmer()
def stemming_tokenizer(str_input):
words = re.sub(r"[^A-Za-z0-9\-]", " ", str_input).lower().split()
words = [porter_stemmer.stem(word) for word in words]
return words
count_vectorizer = CountVectorizer(stop_words='english', tokenizer=boring_tokenizer)
X = count_vectorizer.fit_transform(texts)
print(count_vectorizer.get_feature_names())
Now lets look at the new version of that dataframe.
In [27]:
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
Out[27]:
TF-IDF? What? It means term frequency inverse document frequency! It's the most important thing. Let's look at our list of phrases
If we're searching for the word fish, which is the most helpful phrase?
In [ ]:
pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
Probably the one where fish appears three times.
It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish.
But are all the others the same?
Penny is a fish.
Penny went to the store. Penny ate a bug. Penny saw a fish.
In the second one we spend less time talking about the fish. Think about a huge long document where they say your name once, versus a tweet where they say your name once. Which one are you more important in? Probably the tweet, since you take up a larger percentage of the text.
This is term frequency - taking into account how often a term shows up. We're going to take this into account by using the TfidfVectorizer in the same way we used the CountVectorizer.
In [28]:
from sklearn.feature_extraction.text import TfidfVectorizer
In [29]:
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
Out[29]:
Now our numbers have shifted a little bit. Instead of just being a count, it's the percentage of the words.
value = (number of times word appears in sentence) / (number of words in sentence)
After we remove the stopwords, the term fish is 50% of the words in Penny is a fish vs. 37.5% in It meowed once at the fish, it is still meowing at the fish. It meowed at the bug and the fish..
Note: We made it be the percentage of the words by passing in
norm="l1"- by default it's normally an L2 (Euclidean) norm, which is actually better, but I thought it would make more sense using the L1 - a.k.a. terms divided by words -norm.
So now when we search we'll get more relevant results because it takes into account whether half of our words are fish or 1% of millions upon millions of words is fish. But we aren't done yet!
In [30]:
tfidf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=False, norm='l1')
X = tfidf_vectorizer.fit_transform(texts)
df = pd.DataFrame(X.toarray(), columns=tfidf_vectorizer.get_feature_names())
df
Out[30]:
What's the highest combined? for 'fish' and 'meow'?
In [31]:
# Just add the columns together
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T
Out[31]:
Indices 4 and 6 (numbers 5 and 7) are tied - but meow never even appears in one of them!
It meowed once at the bug, it is still meowing at the bug and the fish
Penny is a fish
It seems like since fish shows up again and again it should be weighted a little less - not like it's a stopword, but just... it's kind of cliche to have it show up in the text, so we want to make it less important.
This is inverse term frequency - the more often a term shows up across all documents, the less important it is in our matrix.
In [32]:
# use_idf=True is default, but I'll leave it in,idf inverse document frequency
idf_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True, norm='l1')
X = idf_vectorizer.fit_transform(texts)
idf_df = pd.DataFrame(X.toarray(), columns=idf_vectorizer.get_feature_names())
idf_df
Out[32]:
Let's take a look at our OLD values, then our NEW values, just for meow and fish.
In [34]:
# OLD dataframe
pd.DataFrame([df['fish'], df['meow'], df['fish'] + df['meow']], index=["fish", "meow", "fish + meow"]).T
Out[34]:
In [33]:
# NEW dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T
Out[33]:
Notice how 'meow' increased in value because it's an infrequent term, and fish dropped in value because it's so frequent.
That meowing one (index 4) has gone from 0.50 to 0.43, while Penny is a fish (index 6) has dropped to 0.40. Now hooray, the meowing one is going to show up earlier when searching for "fish meow" because fish shows up all of the time, so we want to ignore it a lil' bit.
But honestly I wasn't very impressed by that drop.
And this is why defaults are important: let's try changing it to norm='l2' (or just removing norm completely).
In [46]:
# use_idf=True is default, but I'll leave it in
l2_vectorizer = TfidfVectorizer(stop_words='english', tokenizer=stemming_tokenizer, use_idf=True)
X = l2_vectorizer.fit_transform(texts)
l2_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
l2_df
Out[46]:
In [47]:
# normal TF-IDF dataframe
pd.DataFrame([idf_df['fish'], idf_df['meow'], idf_df['fish'] + idf_df['meow']], index=["fish", "meow", "fish + meow"]).T
Out[47]:
In [48]:
# L2 norm TF-IDF dataframe
pd.DataFrame([l2_df['fish'], l2_df['meow'], l2_df['fish'] + l2_df['meow']], index=["fish", "meow", "fish + meow"]).T
Out[48]:
LOOK AT HOW IMPORTANT MEOW IS. Meowing is out of this world important, because no one ever meows.
When someone dumps 100,000 documents on your desk in response to FOIA, you'll start to care! One of the reasons understanding TF-IDF is important is because of document similarity. By knowing what documents are similar you're able to find related documents and automatically group documents into clusters.
For example! Let's cluster these documents using K-Means clustering (check out this gif)
In [39]:
# Initialize a vectorizer
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=boring_tokenizer, stop_words='english')
X = vectorizer.fit_transform(texts) #fit_transform
In [40]:
# KMeans clustering is a method of clustering.
from sklearn.cluster import KMeans
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
Out[40]:
In [41]:
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
In [42]:
results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results
Out[42]:
In [43]:
from sklearn.cluster import KMeans
number_of_clusters = 4
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
Out[43]:
In [44]:
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(number_of_clusters):
top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
In [45]:
results = pd.DataFrame()
results['text'] = texts
results['category'] = km.labels_
results
Out[45]:
In [ ]:
max_features: number of tokens
In [49]:
ax=df.plot(kind='scatter',x='fish',y='penni',alpha=0.25)
ax.set_xlabel("Fish")
ax.set_ylabel("Penny")
Out[49]:
In [ ]:
import matplotlib.pyplot as plt
color_list=['r','b','g','y']
colors = [color_list[i]] for i in df['category']
ax.scatter(df['fish'])
ax.set