Tf-idf Walkthrough for scikit-learn

When I was using scikit-learn extremely handy TfidfVectorizer I had some trouble interpreting the results since they seem to be a little bit different from the standard convention of how the term frequency-inverse document frequency (tf-idf) is calculated. Here, I just put together a brief overview of how the TfidfVectorizer works -- mainly as personal reference sheet, but maybe it is useful to one or the other.

---

Sections

• What are Tf-idfs all about?
• Example data
• Raw term frequency
• Normalized term frequency
• Term frequency-inverse document frequency -- tf-idf
• Inverse document frequency
• Normalized tf-idf
• Smooth idf
• Tf-idf in scikit-learn
• TfidfVectorizer defaults

---

What are Tf-idfs all about?

[back to top]

Tf-idfs are a way to represent documents as feature vectors. Tf-idfs can be understood as a modification of the raw term frequencies (tf); the tf is the count of how often a particular word occurs in a given document. The concept behind the tf-idf is to downweight terms proportionally to the number of documents in which they occur. Here, the idea is that terms that occur in many different documents are likely unimportant or don't contain any useful information for Natural Language Processing tasks such as document classification.

Example data

[back to top]

For the following sections, let us consider the following dataset that consists of 3 documents:

Raw term frequency

[back to top]

First, we will start with the raw term frequency tf(t, d), which is defined by the number of times a term t occurs in a document t

---

Alternative term frequency definitions include the binary term frequency, log-scaled term frequency, and augmented term frequency [1].

---

Using the CountVectorizer from scikit-learn, we can construct a bag-of-words model with the term frequencies as follows:

Based on the vocabulary, the word "and" would be the 1st feature in each document vector in tf, the word "is" the 2nd, the word "shining" the 3rd, etc.

Normalized term frequency

[back to top]

In this section, we will take a brief look at how the tf-feature vector can be normalized, which will be useful later.

The most common way to normalize the raw term frequency is l2-normalization, i.e., dividing the raw term frequency vector $v$ by its length $||v||_2$ (L2- or Euclidean norm).

$$v_{norm} = \frac{v}{||v||_2} = \frac{v}{\sqrt{v{_1}^2 + v{_2}^2 + \dots + v{_n}^2}} = \frac{v}{\big(\sum_{i=1}^n v_i \big)^{\frac{1}{2}}}$$

For example, we would normalize our 3rd document 'The sun is shining and the weather is sweet' as follows:

$\frac{[1, 2, 1, 1, 1, 2, 1]}{2} = [ 0.2773501, 0.5547002, 0.2773501, 0.2773501, 0.2773501, 0.5547002, 0.2773501]$

In scikit-learn, we can use the TfidfTransformer to normalize the term frequencies if we disable the inverse document frequency calculation (use_idf: False and smooth_idf=False):

Term frequency-inverse document frequency -- tf-idf

[back to top]

Most commonly, the term frequency-inverse document frequency (tf-idf) is calculated as follows [1]:

$$\text{tf-idf}(t, d) = \text{tf}(t, d) \times \text{idf}(t),$$

where idf is the inverse document frequency, which we will introduce in the next section.

Inverse document frequency

[back to top]

In order to understand the inverse document frequency idf, let us first introduce the term document frequency $\text{df}(d,t)$, which simply the number of documents $d$ that contain the term $t$. We can then define the idf as follows:

$$\text{idf}(t) = log{\frac{n_d}{1+\text{df}(d,t)}},$$

where
$n_d$: The total number of documents
$\text{df}(d,t)$: The number of documents that contain term $t$.

Note that the constant 1 is added to the denominator to avoid a zero-division error if a term is not contained in any document in the test dataset.

Now, Let us calculate the idfs of the words "and", "is," and "shining:"

Using those idfs, we can eventually calculate the tf-idfs for the 3rd document:

Tf-idf in scikit-learn

[back to top]

The tf-idfs in scikit-learn are calculated a little bit differently. Here, the +1 count is added to the idf, whereas instead of the denominator if the df:

$$\text{idf}(t) = log{\frac{n_d}{\text{df}(d,t)}} + 1$$

We can demonstrate this by calculating the idfs manually using the equation above and comparing the results to the TfidfTransformer output using the settings use_idf=True, smooth_idf=False, norm=None. In the following examples, we will be again using the words "and," "is," and "shining:" from document 3.

Normalized tf-idf

[back to top]

Now, let us calculate the normalized tf-idfs. Our feature vector of un-normalized tf-idfs for document 3 would look as follows if we'd applied the equation from the previous section to all words in the document:

Using the l2-norm, we then normalize the tf-idfs as follows:

And finally, we compare the results to the results that the TfidfTransformer returns.

Smooth idf

[back to top]

Another parameter in the TfidfTransformer is the smooth_idf, which is described as

smooth_idf : boolean, default=True
Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions.

So, our idf would then be defined as follows:

$$\text{idf}(t) = log{\frac{1 + n_d}{1+\text{df}(d,t)}} + 1$$

To confirm that we understand the smooth_idf parameter correctly, let us walk through the 3-word example again:

TfidfVectorizer defaults

[back to top]

Finally, we now understand the default settings in the TfidfTransformer, which are:

• use_idf=True
• smooth_idf=True
• norm='l2'

And the equation can be summarized as
$\text{tf-idf} = \text{tf}(t) \times (\text{idf}(t, d) + 1),$
where
$\text{idf}(t) = log{\frac{1 + n_d}{1+\text{df}(d,t)}}.$

References

[back to top]

[1] G. Salton and M. J. McGill. Introduction to modern information retrieval. 1983.