In this mini-assignment, we will build a simple search index, which we will use later for Boolean retrieval. The assignment tasks are again at the bottom of this document.
In [1]:
Summaries_file = 'data/malaria__Summaries.pkl.bz2'
Abstracts_file = 'data/malaria__Abstracts.pkl.bz2'
In [2]:
import pickle, bz2
from collections import namedtuple
Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )
paper = namedtuple( 'paper', ['title', 'authors', 'year', 'doi'] )
for (id, paper_info) in Summaries.items():
Summaries[id] = paper( *paper_info )
Abstracts = pickle.load( bz2.BZ2File( Abstracts_file, 'rb' ) )
Let's have a look at what the data looks like for our example paper:
In [3]:
Summaries[24130474]
Out[3]:
In [4]:
Abstracts[24130474]
Out[4]:
We'll define some utility functions that allow us to tokenize a string into terms, perform linguistic preprocessing on a list of terms, as well as a function to display information about a paper in a nice way. Note that these tokenization and preprocessing functions are rather naive - you may have to make them smarter in a later assignment.
In [5]:
def tokenize(text):
"""
Function that tokenizes a string in a rather naive way. Can be extended later.
"""
return text.split(' ')
def preprocess(tokens):
"""
Perform linguistic preprocessing on a list of tokens. Can be extended later.
"""
result = []
for token in tokens:
result.append(token.lower())
return result
print(preprocess(tokenize("Lorem ipsum dolor sit AMET")))
In [6]:
from IPython.display import display, HTML
import re
def display_summary( id, show_abstract=False, show_id=True, extra_text='' ):
"""
Function for printing a paper's summary through IPython's Rich Display System.
Trims long author lists, and adds a link to the paper's DOI (when available).
"""
s = Summaries[id]
lines = []
title = s.title
if s.doi != '':
title = '<a href=http://dx.doi.org/%s>%s</a>' % (s.doi, title)
title = '<strong>' + title + '</strong>'
lines.append(title)
authors = ', '.join( s.authors[:20] ) + ('' if len(s.authors) <= 20 else ', ...')
lines.append(str(s.year) + '. ' + authors)
if (show_abstract):
lines.append('<small><strong>Abstract:</strong> <em>%s</em></small>' % Abstracts[id])
if (show_id):
lines.append('[ID: %d]' % id)
if (extra_text != ''):
lines.append(extra_text)
display( HTML('<br>'.join(lines)) )
display_summary(22433778)
display_summary(24130474, show_abstract=True)
We will now create an inverted index based on the words in the abstracts of the papers in our dataset.
We will implement our inverted index as a Python dictionary with terms as keys and posting lists as values. For the posting lists, instead of using Python lists and then implementing the different operations on them ourselves, we will use Python sets and use the predefined set operations to process these posting "lists". This will also ensure that each document is added at most once per term. The use of Python sets is not the most efficient solution but will work for our purposes. (As an optional additional exercise, you can try to implement the posting lists as Python lists for this and the following mini-assignments.)
Not every paper in our dataset has an abstract; we will only index papers for which an abstract is available.
In [7]:
from collections import defaultdict
inverted_index = defaultdict(set)
# This may take a while:
for (id, abstract) in Abstracts.items():
for term in preprocess(tokenize(abstract)):
inverted_index[term].add(id)
Let's see what's in the index for the example term 'network':
In [8]:
print(inverted_index['network'])
We can now use this inverted index to answer simple one-word queries, for example to show all papers that contain the word 'amsterdam':
In [9]:
query_word = 'amsterdam'
for i in inverted_index[query_word]:
display_summary(i)
Your name: ...
Construct a function called and_query that takes as input a single string, consisting of one or more words, and returns a list of matching documents. and_query, as its name suggests, should require that all query terms are present in the documents of the result list. Demonstrate the working of your function with an example (choose one that leads to fewer than 100 hits to not overblow this notebook file).
(You can use the tokenize and preprocess functions we defined above to tokenize and preprocess your query. You can also exploit the fact that the posting lists are sets, which means you can easily perform set operations such as union, difference and intersect on them.)
In [ ]:
# Add your code here
Construct a second function called or_query that works in the same way as and_query you just implemented, but returns documents that contain at least one of the words in the query. Demonstrate the working of this second function also with an example (again, choose one that leads to fewer than 100 hits).
In [ ]:
# Add your code here
In [ ]:
# Add your code here
Given the nature of our dataset, how many of the documents from task 3 do you think are actually about The Who? What could you do to prevent these kind of incorrect results? (You don't have to implement this yet!)
Answer: [Write your answer text here]
Answer: [Write your answer text here]
Submit the answers to the assignment as a modified version of this Notebook file (file with .ipynb extension) that includes your code and your answers via Blackboard. Don't forget to add your name, and remember that the assignments have to be done individually and group submissions are not allowed.