Before you run this demo, you'll need to do a few things:
secure.py in the src/ directory. We've provided a secure.py.example for you to work off! Also, ensure that all the folders you referenced in secure.py exist.DATASET_PATH_BASE in secure.py.
In [1]:
# First, let's get all the imports out of the way...
import gensim.models.word2vec as word2vec
from pprint import pprint
import sys
sys.path.append('../')
sys.path.append('../src/')
import searchbetter.search as search
reload(search)
import searchbetter.rewriter as rewriter
reload(rewriter)
import secure
In [2]:
# Create a search engine that searches over all edX courses.
# Under the hood, this uses Python's Whoosh library to index
# the course data stored in a CSV and then run searches against it.
dataset_path = secure.DATASET_PATH_BASE+'udacity-api.json'
index_path = secure.INDEX_PATH_BASE+'udacity'
# Use `create=False` if you've already made the search engine, `create=True` if this is
# your first time making it. We cache the search indices behind search engines on disk.
### UNCOMMENT THE BELOW IF YOU'RE RUNNING THIS FOR THE FIRST TIME
# udacity_engine = search.UdacitySearchEngine(dataset_path, index_path, create=True)
### COMMENT THE BELOW IF YOU'RE RUNNING THIS FOR THE FIRST TIME
udacity_engine = search.UdacitySearchEngine(dataset_path, index_path, create=False)
# We expose a simple searching API
search_term = "android"
udacity_results = udacity_engine.search(search_term)
print "%d Udacity search results for '%s':" % (len(udacity_results), search_term)
pprint(udacity_results)
print "\n"
# Searching works on bigrams (two-word queries) too!
search_term = "machine learning"
udacity_results = udacity_engine.search(search_term)
print "%d Udacity search results for '%s':" % (len(udacity_results), search_term)
pprint(udacity_results)
Sometimes the plain-vanilla search engine just doesn't cut it. Sometimes search queries don't return enough results. With query rewriting, the search engine looks for semantically related terms to the user's query in addition to the query itself. This helps find more search results, which is particularly useful if the bare query doesn't get any hits.
SearchBetter has two built-in query rewriters: a simple one that uses Wikipedia's Categories API to find similar terms, and a more complex one that uses Google's Word2Vec (a ML similar-word-finding algorithm trained on Wikipedia article dumps) to find similar phrases.
In [3]:
# Query rewriting lets you turn a single search query into
# multiple related queries. You can then search for *all*
# of these queries, which can result in more and more useful
# results than just the original query would give.
# First, a rewriter that uses the Wikipedia category API
# to find terms related to the original term
wiki_rewriter = rewriter.WikipediaRewriter()
term = "socialism"
wiki_rewritten_queries = wiki_rewriter.rewrite(term)
print "Rewrites of '%s' using Wikipedia Categories:" % term
pprint(wiki_rewritten_queries)
print "\n"
# Second, a rewriter that uses Word2Vec to find similar
# words to the entered term. This is a machine learning
# algorithm trained on a large text corpus.
# Prepare the corpus (from Wikipedia) to use for the Word2Vec Rewriter.
corpus = word2vec.LineSentence(secure.DATASET_PATH_BASE + 'wikiclean8')
# Now make the rewriter...
model_path = secure.MODEL_PATH_BASE+'word2vec/word2vec'
## UNCOMMENT the below line if it's your first time making this rewriter
# w2v_rewriter = rewriter.Word2VecRewriter(model_path, create=True, corpus=corpus, bigrams=True)
## UNCOMMENT the below line if you've made the rewriter before
w2v_rewriter = rewriter.Word2VecRewriter(model_path, create=False)
w2v_rewritten_queries = w2v_rewriter.rewrite(term)
print "Rewrites of '%s' using Word2Vec:" % term
pprint(w2v_rewritten_queries)
As we've seen, query rewriters convert one search term into a set of semantically similar ones. Hopefully, by searching for the whole set of terms instead of just one term, we could get more (and more useful) results out of a search engine.
With SearchBetter, you can connect any query rewriter to any search engine and automatically start getting more results.
In [4]:
# Let's plug our two rewriters into the search engine
# to compare the results
# Suppose this is the user's search term
search_term = 'artificial intelligence'
# First, what do we get without rewriting?
udacity_engine.set_rewriter(None)
bare_results = udacity_engine.search(search_term)
print "Without rewriting, %d results for '%s':" % (len(bare_results), search_term)
pprint(bare_results)
print "\n"
# Next, try the Wikipedia rewriter
udacity_engine.set_rewriter(wiki_rewriter)
wiki_rewritten_results = udacity_engine.search(search_term)
print "With Wikipedia Categories rewriting, %d results for '%s':" % (len(wiki_rewritten_results), search_term)
pprint(wiki_rewritten_results)
print "\n"
# Last, try the Word2Vec rewriter
udacity_engine.set_rewriter(w2v_rewriter)
w2v_rewritten_results = udacity_engine.search(search_term)
print "With Word2Vec rewriting, %d results for '%s':" % (len(w2v_rewritten_results), search_term)
pprint(w2v_rewritten_results)
If you want to use SearchBetter's query rewriting and search engine capabilities in your own project, you have two easy options:
search.WhooshSearchEngine. You can find examples in search.py. All you have to specify is how to read over the dataset and put the data into the search engine index.search.GenericSearchEngine and get access to SearchBetter's query rewriting power with no additional work. Here's an example of how you'd wrap a custom search engine in a search.GenericSearchEngine to start taking advantage of our query rewriting functionality.
In [5]:
def prebuilt_black_box_search(term):
# this is an example of a custom, pre-built search engine
# that you can't change or look inside (i.e. it's a black box)
strings = [
"let sleeping dogs lie",
"raining cats and dogs",
"bay of pigs",
"duck duck goose",
"sacred cow",
"llama llama llama",
"poison dart frog"
]
# do a simple text search for the term within the corpus
# of strings
matching_strings = [s for s in strings if term in s]
return matching_strings
class SampleSearchEngine(search.GenericSearchEngine):
"""
A sample search engine that wraps the
`magic_black_box_search` search function.
"""
def __init__(self):
super(SampleSearchEngine, self).__init__()
def single_search(self, term):
return prebuilt_black_box_search(term)
sample_engine = SampleSearchEngine()
# first do a search without rewriting
term = "dog"
print "Before rewriting:"
pprint(sample_engine.search(term))
print "\n"
# now add a rewriter and search again
# you'll find that we get more results!
sample_engine.set_rewriter(w2v_rewriter)
print "After Word2Vec rewriting:"
pprint(sample_engine.search(term))