Demo for EMSE 6992, "Machine Learning for Analytics" on 1/30/19.
This notebook shows examples of using command-line tools for working with Twitter data.
For this exercise, we're using a dataset with 2,000 tweets collected 01/01/2019 from the Twitter sample stream API. Filename: sample.json
How many tweets are in this file? Look at the first tweet.
In [1]:
!wc -l sample.json
head with a number flag (e.g. -5) determines the number of lines to show. Here's the first line, which is also the first line in line-oriented JSON.
In [2]:
!head -1 sample.json
View the first tweet in the file using jq.
In [3]:
!head -1 sample.json | jq '.'
Choose multiple fields to extract from all tweets, using jq and the -c flag for compact output.
The following examples uses head to only show a few lines and truncate the output here, but you can also use cat to process the whole file.
In [1]:
!head sample.json | jq -c '[.id_str, .text, .user.screen_name, .created_at]'
For filtering and output, it is helpful to transform the date into a more uesful format. Let's look at the original created_at date and transform it into an ISO 8601 date.
In [2]:
!head sample.json | jq -c '[.created_at, (.created_at | strptime("%A %B %d %T %z %Y") | todate)]'
Look at all of the tweets in particular language, in this case Spanish ("es").
In [3]:
!head sample.json | jq -c 'select(.lang | contains("es")) | [.text]'
Create a CSV with a subset of the fields.
In [7]:
!cat sample.json | jq -r '[.id_str, .created_at, .text] | @csv' > tweets.csv
In [8]:
!head -20 tweets.csv
Note that that newlines in the tweet text break the CSV format, which should only have line breaks at the end of the row. Let's fix problems with the text field in the tweets by replacing the \n newline character with a space.
In [4]:
!cat sample.json | jq -r '[.id_str, (.text | gsub("\n";" "))] | @csv' > tweets.csv
In [5]:
!head -20 tweets.csv
Hashtags are in nested JSON, so let's flattening the hashtags into a semi-colon delimited list.
In [9]:
!head -200 sample.json | jq -c 'select(.entities.hashtags | length >= 1) | [([.entities.hashtags[].text] | join(";"))]'