Datasets: Downloading Data from Google Trends

28th May 2014

Neil Lawrence

This data set collection was inspired by a ipython notebook from sahuguet which made queries to google trends and downloaded the results. We've modified the download to cache the results of a query: making multiple calls to the google API results in a block due to terms of service violations, cacheing the data locally prevents this happening.


In [1]:
import GPy
%matplotlib inline

In [2]:
# calling without arguments uses the default query terms
data = GPy.util.datasets.google_trends()


Reading cached data for google trends. To refresh the cache set 'refresh_data=True' when calling this function.
Query terms:  big data, data science, machine learning

The default query terms are 'big data', 'data science' and 'machine learning'. The dictionary returned from the call contains the standard 'X' and 'y' keys that are ready to be used in the GPy toolkit as inputs to the Gaussian process. In this case the 'X' variables are the time (first column) and an index representing the query.


In [3]:
print data['X'][284, :]


[34  2]

So the 284th element of X contains is the 34th time point of the query term 2, which in this case is the 34th time point of the 'machine learning' time series. The value of the time series at that point is given by the corresponding row of Y


In [4]:
print data['Y'][284, :]


[ 20.]

The dictionary also contains a pandas data frame of the trend data, which is in line with what sahuguet originally returned.


In [5]:
data['data frame'].describe()


Out[5]:
big data data science machine learning
count 125.000000 125.000000 125.000000
mean 20.184000 11.856000 21.472000
std 25.836766 3.252056 5.864843
min 4.000000 7.000000 14.000000
25% 7.000000 10.000000 17.000000
50% 8.000000 11.000000 20.000000
75% 17.000000 13.000000 24.000000
max 100.000000 21.000000 43.000000

And we can plot the trends data to see what the effect is.


In [6]:
data['data frame'].plot()


Out[6]:
<matplotlib.axes.AxesSubplot at 0x10f8117d0>

Dogs, Cats and Rabbits

Another data set we might consider downloading from google trends is different pets. Below we consider cats, dogs and rabbits.


In [7]:
data = GPy.util.datasets.google_trends(['cats', 'dogs', 'rabbits'])
data['data frame'].set_index('Date', inplace=True)
data['data frame'].plot()


Reading cached data for google trends. To refresh the cache set 'refresh_data=True' when calling this function.
Query terms:  cats, dogs, rabbits
Out[7]:
<matplotlib.axes.AxesSubplot at 0x10f81ee90>

Here we've plotted the data in the same manner as sahuguet suggested in his original notebook, using the plotting facility of pandas.


In [7]: