In this notebook we'll take a quick look at the section of the Twitter Rest API that deals with trending terms:
We'll work with the Tweepy library's support for these, explore the calls, and sketch out some possibilities.
First some library imports and user key setup. This assumes you've arranged for the necessary app keys at apps.twitter.com.
In [74]:
from collections import Counter
import os
from pprint import pprint
import tweepy
In [58]:
c_key = os.environ['CONSUMER_KEY']
c_secret = os.environ['CONSUMER_SECRET']
a_token = os.environ['ACCESS_TOKEN']
a_token_secret = os.environ['ACCESS_TOKEN_SECRET']
auth = tweepy.OAuthHandler(c_key, c_secret)
auth.set_access_token(a_token, a_token_secret)
api = tweepy.API(auth)
In [59]:
trends_available = api.trends_available()
In [60]:
len(trends_available)
Out[60]:
In [61]:
trends_available[0:3]
Out[61]:
We see from this (and from their docs) that Twitter uses Yahoo WOEIDs to define place names. How they arrived at this list of 467 places is not clear. It's a mix of countries and cities, at least:
In [62]:
sorted([t['name'] for t in trends_available])[:25]
Out[62]:
Hmm, let's see exactly what classification levels they have in this set:
In [76]:
c = Counter([t['placeType']['name'] for t in trends_available])
In [77]:
c
Out[77]:
'Unknown'??
In [78]:
unk = [t for t in trends_available if t['placeType']['name'] == 'Unknown']
In [79]:
unk
Out[79]:
Ah, Soweto is a township, not a city, and Al Ahsa is a region.
Moving along, we can dig into specifc places one at a time:
In [8]:
ams = [t for t in trends_available if t['name'] == 'Amsterdam'][0]
In [64]:
ams
Out[64]:
In [63]:
trends_ams = api.trends_place(ams['woeid'])
In [65]:
trends_ams
Out[65]:
In [68]:
trends_ams[0]['trends']
Out[68]:
In [71]:
[(t['name'], t['tweet_volume']) for t in trends_ams[0]['trends']]
Out[71]:
There's also a way to fetch the list without hashtags:
In [72]:
trends_ams = api.trends_place(ams['woeid'], exclude='hashtags')
In [73]:
[(t['name'], t['tweet_volume']) for t in trends_ams[0]['trends']]
Out[73]:
In [36]:
def getloc(name):
try:
return [t for t in trends_available if t['name'].lower() == name.lower()][0]
except:
return None
In [42]:
def trends(loc, exclude=False):
return api.trends_place(loc['woeid'], 'hashtags' if exclude else None)
In [46]:
def top(trends):
return sorted([(t['name'], t['tweet_volume']) for t in trends[0]['trends']], key=lambda a: a[1] or 0, reverse=True)
In [80]:
tok = getloc('tokyo')
In [81]:
trends_tok = trends(tok, True)
In [82]:
top(trends_tok)
Out[82]:
In [83]:
alg = getloc('Algeria')
In [84]:
trends_alg = trends(alg)
In [90]:
top(trends_alg)
Out[90]:
Weird, there's an L-to-R issue rendering the output here:
In [92]:
top(trends_alg)[1][0]
Out[92]:
Just for fun let's compare Algiers with Algeria.
In [102]:
algs = getloc('Algiers')
In [103]:
trends_algs = trends(algs)
In [104]:
top(trends_algs)
Out[104]:
Very similar. How about Chicago vs Miami?
In [119]:
chi = getloc('Chicago')
In [112]:
trends_chi = trends(chi)
In [114]:
chi
Out[114]:
In [115]:
top(trends_chi)[:10]
Out[115]:
In [120]:
mia = getloc('Miami')
In [121]:
mia
Out[121]:
In [122]:
trends_mia = trends(mia)
In [123]:
top(trends_mia)[:10]
Out[123]:
In [124]:
world = getloc('Worldwide')
In [125]:
trends_world = trends(world)
In [127]:
top(trends_world)[:25]
Out[127]:
There are specific javascript calls for fetching a user's geolocation through a browser (see Using Geolocation via MDN). With a tool like that in place, you could fetch the user's lat and long and send it to the trends/closest Twitter API call:
In [95]:
closeby = api.trends_closest(38.8860307, -76.9931073)
In [99]:
closeby[0]
Out[99]:
In [101]:
top(trends(closeby[0]))
Out[101]:
Hmm, that parentid attribute might be useful.
In [135]:
[t['name'] for t in trends_available if t['parentid'] == 23424977]
Out[135]:
In [130]:
[t['name'] for t in trends_available if t['parentid'] == world['woeid']]
Out[130]:
No Liechtenstein? Georgia? Belize? Hmm.
The Twitter API calls have the following rate limits (as of October 17, 2016):
GET trends/available - 15/15 minGET trends/place - 15/15 minGET trends/closest - 15/15 minThe key issue here is trends/place. trends/available probably doesn't change very often. trends/closest would only vary if the user is on the move, and even then, only slowly. But to track trends at more than one level on a minute-over-minute basis would simply not be possible because we are limited to one call per minute on trends/place.
Practically speaking, we could do:
Given the practicality of setting things up and allowing a little buffer, it probably makes sense to check three places every five minutes. The default scenario could be check (worldwide, country, closest place). A config parameter could figure this out for you if set by default, or a little UI element could help choose places. For example, if a continuing event users wanted to track occurred in Omaha, you could have an option to switch one of the ongoing trend trackers to get Omaha every five minutes, or to turn off all the defaults and fetch trends for Omaha once every minute.