Food is an important part of everybody's life. Not only does food influence how we look and feel, agriculture also has a profound impact on ecosystems, economies, and politics. Yet when we go to the grocery store, it can be difficult to really know exactly what we're purchasing and where it comes from.
Inspired by this problem, a few of us decided to build an application that provides information on a packaged food product based on an image taken with a smartphone. In a future blog post, we will share what we built and how we built it. In this notebook, however, we delve deeper into the actual implementation.
This notebook is divided into 5 main parts:
Below are two screenshot of what the app looks like. The first image is a photo taken on the phone, and the second is matched results. Pretty good! To learn what it takes to build an app like this, read on!
First we download the SFrame containing the openfood dataset, which is a listing of packaged food items from around the world. Although the dataset contains about 30000 package foods, we filter for ones from english speaking countries. This ends up giving us about 2000 packaged food items in the SFrame.
In [1]:
import graphlab
import base64
In [2]:
openfood_sf = graphlab.SFrame('https://static.turi.com/datasets/food/filtered_openfood_sf')
Inspect the products in the dataset
In [3]:
openfood_sf['product_name']
Out[3]:
The Openfood dataset contains some repeated product names(for instance, 'Intensely Fruity Christmas Pudding'), so we use the deduplication toolkit to remove copies. Here, we only want to deduplicate names that are very similar. Therefore, we set the radius parameter to be fairly small. The radius is the maximum distance from each point to another so that they are still considered duplicates.
In [4]:
dedup = graphlab.nearest_neighbor_deduplication.create(openfood_sf, features=['product_name'],radius=0.25)
We select one element from each entity group to be our cleaned dataset.
In [5]:
dedup_sf = dedup['entities'].groupby(key_columns="__entity", operations = {'row_number' : graphlab.aggregate.SELECT_ONE('row_number')})
openfood_sf = openfood_sf.add_row_number('row_number').filter_by(dedup_sf['row_number'], 'row_number')
Now, there are many fewer duplicates.
In [6]:
openfood_sf['product_name']
Out[6]:
We start by using the autotagger to find similar foods within the dataset, based on textual information like ingredients and category. The autotagger toolkit pre-processes the query and tags to extract character 4-grams, unigrams, and bigrams as features, then employs the nearest neighbor toolkit to do the similarity search with a weighted jaccard distance. In this case both queries and tags are entries in the dataset, and the result is a matching between each entry and other similar entries.
We start by concatenating several features to create the query and tag.
In [7]:
openfood_sf['product_features'] = openfood_sf['product_name'] + ', ' + openfood_sf['generic_name'] + ' ' + openfood_sf['categories'] + ' ' + openfood_sf['packaging'] + ' ' + openfood_sf['brands'] + ' ' + openfood_sf['labels'] + ' ' + openfood_sf['ingredients_text'] + ' ' + openfood_sf['allergens'] + ' ' + openfood_sf['additives']
Next, we set the reference set equal to the tags. Now we'll a set of food items that are most similar to each item in the openfood database.
In [8]:
openfood_sf['product_tags'] = openfood_sf['product_features']
tagger = graphlab.autotagger.create(openfood_sf, tag_name='product_tags')
Now we query the tagger. Here, we choose k=4 so we can retrieve 4 tags, or nearest neighbors. Note that the nearest tag will be the same as the query since the query set and tag set are identical.
In [9]:
openfood_similar_products = tagger.tag(openfood_sf, query_name='product_features', k=4)
Remove tags that equal the query , and do some operations to make the results human-reader friendly.
In [10]:
openfood_similar_products = openfood_similar_products[openfood_similar_products['product_features'] != openfood_similar_products['product_tags']]
openfood_similar_products['product_name'] = openfood_similar_products['product_tags'].apply(lambda x: x.split(',')[0])
openfood_similar_products_groupby = openfood_similar_products.groupby(key_columns='product_features_id', operations={'similar_foods': graphlab.aggregate.CONCAT('product_name')})
openfood_sf = openfood_sf.add_row_number('row_id')
openfood_sf = openfood_sf.join(openfood_similar_products_groupby, how='left', on={'row_id':'product_features_id'})
openfood_sf = openfood_sf.fillna('similar_foods', ['','',''])
Inspect the results! They look good!
In [11]:
openfood_sf.select_columns(['product_name','similar_foods']).unpack('similar_foods')
Out[11]:
We will be using deep visual features to match our personal photos of food to the catalog provided by openfood. In order to do that, we need to load in our pre-trained ImageNet neural network model to be used as a feature extractor, and extract features from the images in the dataset. To learn more about feature extraction, read this blog post. Note: This codeblock assumes that you do not have a GPU, and loads pre-extracted features. If you want to extract features yourself, uncomment the lines below.
In [12]:
visual_features = graphlab.SFrame()
visual_features['visual_features'] = graphlab.SArray('https://static.turi.com/datasets/food/openfood_extracted_features')
visual_features = visual_features.add_row_number('row_number').filter_by(dedup_sf['row_number'], 'row_number')
openfood_sf['visual_features'] = visual_features['visual_features']
#pretrained_model = graphlab.load_model('http://s3.amazonaws.com/GraphLab-Datasets/deeplearning/imagenet_model_iter45')
#openfood_sf['visual_features'] = pretrained_model.extract_features(openfood_sf)
Now we need to build the Nearest Neighbors model which we will search with our own personal photos of food items. Note that we previously were were using the autotagger, while here we are using the Nearest Neighbor Model. This is because previously we were measuring similarity between text features, and the autotagger handles the featurization of text data. However, with images, we've already extracted deep features, and we can directly use the Nearest Neighbor Model.
In [13]:
m = graphlab.nearest_neighbors.create(openfood_sf, features=['visual_features'])
In [14]:
peanut_butter = graphlab.SFrame({'image':[graphlab.Image('https://static.turi.com/datasets/food/pb.jpg')]})
pretrained_model = graphlab.load_model('http://s3.amazonaws.com/GraphLab-Datasets/deeplearning/imagenet_model_iter45')
peanut_butter['image'] = graphlab.image_analysis.resize(peanut_butter['image'], 256, 256, 3)
peanut_butter['visual_features'] = pretrained_model.extract_features(peanut_butter)
Let us take a look at the query image. It is clearly Jif peanut butter.
In [16]:
peanut_butter.show()
Out[16]:
And now we query the model. This finds the nearest items in the catalog to our photo.
In [17]:
pb_ans = m.query(peanut_butter)
Now, we extract the nearest neighbors out of the original SFrame with a join the row id, and use GraphLab Canvas(which opens in a browser window) to explore similar items foods to the JIF. For easiest viewing, switch to the Table view. Sure enough, JIF peanut butter is there! It appears to have similar items that are other peanut butters!
In [18]:
pb_ans.join(openfood_sf,on={"reference_label":"row_id"}).select_columns(['image','similar_foods']).show()
Out[18]:
In [18]:
env = graphlab.deploy.Ec2Config(region='us-west-1',
instance_type='m3.large',
aws_access_key_id=YOUR_ACCESS_KEY,
aws_secret_access_key=YOUR_SECRET_KEY)
In [19]:
deployment = graphlab.deploy.predictive_service.create('food-app-notebook',env, 's3://gl-internal-test/food_app/predictive_service_notebook')
Select useful columns.
In [20]:
openfood_sf_ps = openfood_sf.select_columns(['image_url', 'product_name', 'generic_name', 'proteins_100g', 'fat_100g', 'carbohydrates_100g', 'energy_100g','similar_foods','row_id'])
Function to construct GraphLab Image from bytestring
In [21]:
def image_from_bytestring(image_data):
import cStringIO as StringIO
from PIL import Image as _PIL_image
decoded_image_data = base64.b64decode(image_data)
stream = StringIO.StringIO(decoded_image_data)
pil_img = _PIL_image.open(stream)
width = pil_img.size[0]
height = pil_img.size[1]
format_to_num = {'JPG': 0, 'PNG': 1, 'RAW': 2}
if pil_img.mode == 'L':
channels = 1
elif pil_img.mode == 'RGB':
channels = 3
else:
channels = 4
format_enum = format_to_num['RAW']
if(pil_img.format == 'JPEG' or pil_img.format == 'JPG'):
format_enum = format_to_num["JPG"]
if(pil_img.format == "PNG"):
format_enum = format_to_num["PNG"]
image_data_size = len(decoded_image_data)
#setting the appropriate attributes
img = graphlab.Image()
img._image_data = decoded_image_data
img._height = height
img._width = width
img._channels = channels
img._format_enum = format_enum
img._image_data_size = image_data_size
return img
Define function which queries nearest neighbors model to return matching foods in the database and products similar to them.
In [22]:
import base64
from graphlab.deploy import required_packages
@required_packages(["Pillow==2.7.0"])
def match_image(image_bytestring):
image_sf = graphlab.SFrame({'image' : [image_from_bytestring(image_bytestring)]})
image_sf['image'] = graphlab.image_analysis.resize(image_sf['image'], 256, 256, 3)
image_sf['visual_features'] = pretrained_model.extract_features(image_sf)
ans = m.query(image_sf)
ret_sf = ans.join(openfood_sf_ps,on={"reference_label":"row_id"})
return ret_sf
Add the match_image function to the predictive service
In [23]:
deployment.add('get_similar_food', match_image, description='Get closest neighbors to image in openfood facts dataset')
Issue a local test query
In [24]:
deployment.test_query('get_similar_food', image_bytestring = base64.b64encode(str(graphlab.Image('https://static.turi.com/datasets/food/pb.jpg')._image_data)))
Out[24]:
It worked, so deploy the change and perform a real query!
In [28]:
deployment.apply_changes()
deployment.query('get_similar_food', image_bytestring = base64.b64encode(str(graphlab.Image('https://static.turi.com/datasets/food/pb.jpg')._image_data)))
Out[28]:
Don't forget to the terminate the service once you are done!
In [29]:
deployment.terminate_service()
Go ahead and take some food photos at the grocery store, and try this out for yourself!