TODO: update this
This iPython notebook walks you through reading data from a Fluxtream data channel, computing a function on it, and uploading a new computed channel back to Fluxtream. This particular example reads data from a relative humidity channel, such as those from the Netatmo Humidity field, and writes back a value that I'm playing with in the hope may give some insight into how conducive that point in time and space was for growing mold.
The computation itself is fairly naive, and is just based on the EPA Mold Guide saying "keep indoor humidity below 60 percent (ideally between 30 and 50 percent) relative humidity". I have no idea how the risk scales with RH. I'm arbitrarily treating under 55% as 0, and treating values from 55% to 100% as linearly increasing. That's probably not correct, but should at least highlight times to be more or less concerned about, and help see if the dehumidifier in the basement is helping vs. last year.
The real value of this notebook, I hope, is to provide an example of how to read data from a Fluxtream datastore channel, do something to it, and write it back. If you have a Netatmo, you can run the netatmo-fluxtream-gateway notebook to upload your channels to try this on. Otherwise, if you email me at info@fluxtream.org I can add you as a buddy to access the Netatmo channels uploaded to the test account (Guest ID=1). Or you can just modify the channel names and function to do something interesting to your own data channels.
If you are new to iPython notebooks, here is the main IP[y] website. You will need to install python and iPython notebook on your local system, run a local ipython kernel, and install a local copy of this notebook to be able to execute and modify the code below. Install instructions are here. On osx systems, you can start the server by going to Terminal and calling 'ipython notebook'. This will start a local web server and open a an IP[y] page talking to it in a web browser. Within the IP[y] page, you can open a saved iPython notebooy by going to File/Open.
Once you have IP[y] generally working on your system, here's a brief intro in how to use it:
When a given cell is executed, it may print output which appears below the cell, and the cursor will continue to the next cell. If the next cell is tall, you might need to scroll back up to see the previous cell's output.
Each cell in this notebook pertains to a particular step in the process, topic, or action requiring your input and contains comments at the top saying what it's about and what you should do.
Cells that require entry of sensitive information, such as passwords, start with a phrase like "Execute and fill in the fields below". These generally create entry forms in the output area below the cell that you need to fill in. Generally, the cell will clear the sensitive input boxes after clicking the button. It may also print out suggestions about how you could set up a new cell for future use if you're confident other's won't see your copy of the notebook.
Cells that require customization for your own setup start with "Modify". These include cells where you configure which channels you want to process and what you want to call the resulting data. These require some thought.
Cells that define functions or do other things that don't require user input or modification generally just start with "Execute". These can just be executed without much consideration, though you may want to go back later to understand or modify them.
Please enjoy, tinker, modify, etc. Feel free to contact info@fluxtream.org if you have questions.
Note that uploading data multiple times to a given device and channel with identical time values each time will safely overwrite the previous values. However, there is no API or user interaction component in Fluxtream that allows the deletion of a previously-uploaded device or channel, and you can't delete data points already uploaded to a given channel. If you create device names or channel names, or upload data at incorrect timepoints within a given channel, and later regret it, please send the info about your situation, including your Fluxtream username, guest ID, and the details of which devices and or channels you want deleted to info@fluxtream.org. You can get your Guest ID by doing the step below to set up your Fluxtream credentials and looking at the value of fluxtream_guest_id. Also note that the Fluxtream upload API cannot currently handle empty cells within the data array used in an upload call. I'm hoping to fix this in the future.
In [1]:
# Execute this cell to define the functions for calling the Fluxtream upload API for the
# credentials entered below
import base64, json, subprocess, urllib, urllib2, csv
# By default, the upload function will send data to the main server at fluxtream.org.
# If you want to have this use a different fluxtream server, change it here
# and make sure the username and password entered below are valid on that server.
fluxtream_server = 'fluxtream.org'
def fluxtream_authenticate(username, password):
global fluxtream_server, fluxtream_username, fluxtream_password, fluxtream_guest_id
fluxtream_username = username
fluxtream_password = password
fluxtream_guest_id = None
try:
req = urllib2.Request('http://%s/api/guest' % fluxtream_server,
headers = fluxtream_headers())
response = json.loads(urllib2.urlopen(req).read())
fluxtream_guest_id = int(response['id'])
except urllib2.HTTPError, e:
raise Exception('Failed to authenticate (server says %s)' % e)
print 'Verified credentials for user %s on %s (guest ID=%d)' % (fluxtream_username, fluxtream_server, fluxtream_guest_id)
return True
def fluxtream_headers():
auth = base64.encodestring('%s:%s' % (fluxtream_username, fluxtream_password)).replace('\n', '')
return {'Authorization': 'Basic ' + auth}
def fluxtream_confirm_authenticated():
global fluxtream_guest_id
if fluxtream_guest_id == None:
raise Exception('Need to enter Fluxtream credentials before connecting.')
def fluxtream_upload(dev_nickname, channel_names, data):
global fluxtream_server, fluxtream_username, fluxtream_password
# Make sure we have some data to send
if data == None or len(data)<1:
print 'Nothing to upload to %s %s' % (dev_nickname, channel_names)
return
# Make sure we have fluxtream credentials set properly
if not('fluxtream_server' in globals() and
'fluxtream_username' in globals() and
'fluxtream_password' in globals()):
raise
# Send to BodyTrack upload API, documented at
# https://fluxtream.atlassian.net/wiki/display/FLX/BodyTrack+server+APIs#BodyTrackserverAPIs-Storingdata
cmd = ['curl', '-v']
cmd += ['-u', '%s:%s' % (fluxtream_username, fluxtream_password)]
cmd += ['-d', 'dev_nickname=%s' % dev_nickname]
cmd += ['-d', 'channel_names=%s' % json.dumps(channel_names)]
cmd += ['-d', 'data=%s' % json.dumps(data)]
cmd += ['https://%s/api/bodytrack/upload' % fluxtream_server]
print 'Uploading %d data points to %s\'s account on server %s, device %s, channels %s' % (len(data),
fluxtream_username,
fluxtream_server,
dev_nickname,
channel_names)
# If you're having trouble debugging this function, uncomment the following two print statements
# to see the exact curl command and result string
#print ' Cmd=%s' % (cmd)
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
try:
response = json.loads(result_str)
if response['result'] != 'OK':
raise Exception('Received non-OK response %s while trying to upload to %s' % (response, dev_nickname))
print 'Upload to %s %s (%d rows, %d to %d) succeeded' % (dev_nickname, channel_names, len(data), data[0][0], data[-1][0])
except:
print "Attempt to upload to %s as user %s failed. Check that your credentials are ok" % (fluxtream_server,
fluxtream_username)
print "Server returned response: %s" % (result_str)
raise
# To get your own data, pass in the global fluxtream_guest_id which is computed
# in setup_fluxtream_credentials() when you execute the Fluxtream login cell.
# To get a buddy's data, you first need to figure out what their Guest ID is.
# This will show up in the Chrome developer console in tile requests when you
# look at their data in the timeline or BodyTrack app.
# For example, if the test account is my buddy, I would select
# 'View test test's data' from the upper right
# hand menu, turn on developer tools, and go to the Fluxtream
# timeline tab. In the developer tools' network tab I would
# see fetches that look like:
# 7.21370.json
# /api/bodytrack/tiles/1/BodyMedia.activityType
# The value between 'tiles' and the device_name.channel_name is
# that account's Guest ID. In that case, I would call
# fluxtream_get_sources_list with an arg of 1.
def fluxtream_get_sources_list(guest_id):
global fluxtream_server, fluxtream_username, fluxtream_password
# Make sure we have fluxtream credentials set properly.
if not('fluxtream_server' in globals() and
'fluxtream_username' in globals() and
'fluxtream_password' in globals()):
raise Exception("Need to enter Fluxtream credentials. See above.")
# Send to BodyTrack upload API, documented at
# https://fluxtream.atlassian.net/wiki/display/FLX/BodyTrack+server+APIs#BodyTrackserverAPIs-Storingdata
cmd = ['curl', '-v']
cmd += ['-u', '%s:%s' % (fluxtream_username, fluxtream_password)]
cmd += ['https://%s/api/bodytrack/users/%d/sources/list' % (fluxtream_server, guest_id)]
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
try:
response = json.loads(result_str)
print 'Read of sources list for guest_id=%d succeeded' % (guest_id)
return response
except:
print "Attempt to upload to %s as user %s failed. Check that your credentials are ok" % (fluxtream_server,
fluxtream_username)
print "Server returned response: %s" % (result_str)
raise
def fluxtream_get_device_names(sources_list):
device_names = []
for dev in sources_list['sources']:
device_names.append(dev['name'])
return device_names
def fluxtream_get_device_info(device_name, sources_list):
for dev in sources_list['sources']:
if(dev['name'] == device_name):
return dev
return None
def fluxtream_get_channel_names(device_name, sources_list):
dev_info = fluxtream_get_device_info(device_name, sources_list)
channel_names = []
for channel in dev_info['channels']:
channel_names.append(channel['name'])
return channel_names
def fluxtream_get_channel_info(device_name, channel_name, sources_list):
dev_info = fluxtream_get_device_info(device_name, sources_list)
# Check to make sure that we found info for the requested device.
# If not, return None
if not dev_info:
return None
for channel_info in dev_info['channels']:
if(channel_info['name'] == channel_name):
return channel_info
return None
# Takes a guest_id, an array of <device_name>.<channel_name> strings, and a time range and returns a CSV reader.
# Iterate over the rows using reader.next(), which returns a row array with entries corresponding to
# Epoch, [dev_ch_names]
# Where Epoch is the epoch timestamp (aka unixtime) for the values in the row, and the i+1'th column of the row
# corresponds to the channel in dev_ch_names[i]
# See comment on fluxtream_get_sources_list for info about how to choose the value for guest_id
def fluxtream_get_csv(guest_id, dev_ch_names, start_time, end_time):
global fluxtream_server
fluxtream_confirm_authenticated()
# Send to BodyTrack upload API, documented at
# https://fluxtream.atlassian.net/wiki/display/FLX/BodyTrack+server+APIs#BodyTrackserverAPIs-Storingdata
# Need to convert the dev_ch_names array into json and URL encode it to create the channels arg
# TODO: how do we confirm that dev_ch_names is in fact an array?
ch_spec_str = json.dumps(dev_ch_names)
ch_spec_str = urllib.quote(ch_spec_str)
url = ('https://%s/api/bodytrack/exportCSV/%d/fluxtream-export-from-%d-to-%d.csv?channels=%s' %
(fluxtream_server, guest_id, int(start_time), int(end_time), ch_spec_str))
try:
req = urllib2.Request(url, headers = fluxtream_headers())
result_str = urllib2.urlopen(req).read()
except urllib2.HTTPError, e:
raise Exception('Failed to authenticate (server says %s)' % e)
# If the API call worked, result_str should be a CSV file
# with the first line a header consisting of EpochTime, [dev_ch_names]
# Create a CSV reader that iterates over the lines of the response
csv_reader = csv.reader(result_str.splitlines(), delimiter=',')
header = csv_reader.next()
# Do some checks to make sure we got something reasonable
if len(header) != len(dev_ch_names)+1:
raise Exception("Expected header for CSV export of %s to contain %d columns, but only found %d. Please double check that dev_ch_names are all valid" % (dev_ch_names, len(dev_ch_names)+1, len(header)))
# Check the columns are what we expect
for i in range(0,len(dev_ch_names)):
if(dev_ch_names[i] != header[i+1]):
raise Exception("Expected column %d of CSV header to be %s, but found %s instead. Please double check that dev_ch_names are all valid" % (i+1, dev_ch_names[i], header[i+1]))
# At this point, we can be confident that the columns map to Epoch, [dev_ch_names] as expected.
# Return the csv reader. Iterate over the rows using reader.next()
return csv_reader
In [2]:
%%HTML
Fluxtream username: <input id="fluxtream_username" type="text"></input><br>
Fluxtream password: <input id="fluxtream_password" type="password"></input><br>
<button id = "fluxtream_authenticate">Authenticate</button> <span id="fluxtream_auth_result"></span>
<script>
// Send username and password to python
function fluxtream_authenticate(username, password, callback) {
cmd = 'fluxtream_authenticate(' + JSON.stringify(username) + ',' +
JSON.stringify(password) + ')'
console.log(cmd);
function cb(msg) {
console.log(msg);
console.log(msg.content.data['text/plain'] == 'True');
callback(msg.content.data['text/plain'] == 'True')
}
IPython.notebook.kernel.execute(cmd,
{iopub: {output: cb}}, {silent: false});
}
$('#fluxtream_authenticate').click(function() {
$('#fluxtream_auth_result').text('...');
fluxtream_authenticate($('#fluxtream_username').val(),
$('#fluxtream_password').val(),
function(success) {
$('#fluxtream_auth_result').text(success ? 'Success' : 'Failed');
});
});
</script>
In [3]:
from numpy import *
from scipy import *
from pylab import *
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.signal
def plot_hrv(ts_col, rr_col):
duration = ts_col[-1] - ts_col[0]
print 'Duration %g seconds' % duration
# Resample to 10 hz
nsamples = int(round(duration * 10))
nsamples = int(round(nsamples / 1024)) * 1024
print 'Resampling to %d samples' % nsamples
rr = scipy.signal.resample(rr_col, nsamples, t=ts_col)[0]
NFFT=2048
fft_samples = 1000
noverlap = int(round(NFFT - (nsamples / fft_samples)))
plt.figure().set_size_inches(22, 9)
plt.subplot2grid((4,1), (0, 0), rowspan=2)
# colormaps shown here: http://matplotlib.org/examples/color/colormaps_reference.html
(spec, freqs, times, image) = specgram(rr, Fs=10, NFFT=NFFT, noverlap=noverlap, cmap=cmap)
plt.ylim([0,.62])
# Extract "HF", 0.15 Hz to 0.4 Hz
hf = [0] * len(times)
lf = [0] * len(times)
vlf = [0] * len(times)
lf_over_hf = [0] * len(times)
for (i, t) in enumerate(times):
for (j, freq) in enumerate(freqs):
if 0.15 <= freq and freq <= 0.4:
hf[i] += spec[j, i]
if 0.04 <= freq and freq <= 0.15:
lf[i] += spec[j, i] / 2.0
if 0.00 < freq and freq <= 0.04:
vlf[i] += spec[j, i] / 200.0
lf_over_hf[i] = lf[i] / hf[i] / 10.0
plt.subplot2grid((4,1), (2, 0))
plt.plot(times, hf, label='hf')
plt.plot(times, lf, label='lf / 2')
legend()
plt.ylim(0,0.5)
plt.subplot2grid((4,1), (3, 0))
plt.plot(times, vlf, label='vlf / 200')
plt.plot(times, lf_over_hf, label='lf / hf / 10')
legend()
plt.ylim(0,0.5)
In [4]:
# iframe_with_source wraps HTML+javascript into an iframe, to avoid DOM and javascript conflicts
from IPython.display import HTML
import json
def iframe_with_source(source, height='200px'):
name = 'iframe-%d' % get_ipython().execution_count
source = json.dumps(source).replace('</script', '</scr"+"ipt')
template = """
<iframe id="%s" style="width:100%%; height:%s"></iframe>
<script>
document.getElementById('%s').srcdoc = %s;
</script>
"""
# Fill in the %s slots with id, width, height, and the HTML source
return HTML(template % (name, height, name, source))
In [5]:
src = """
<html>
<head>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://api.cmucreatelab.org/timeline-chart-0.1/grapher2.nocache.js"></script>
<style type="text/css">
body {
font-family: "Gill Sans Light", Verdana, Arial, sans-serif !important;
font-size: 10pt !important;
}
</style>
<script>
var series = [];
var dateAxis;
var fluxtream_credentials = null;
var channels = [
{
channel:'PolarStrap.BeatSpacing',
styles: [
{
type: "line",
lineWidth: 1,
show: true,
color: "#0000ff"
}
],
highlight: {
lineWidth: 1,
styles: [
{
show: true,
type: "value",
fillColor: "#ff0000",
marginWidth: 10,
font: "7pt Helvetica,Arial,Verdana,sans-serif",
verticalOffset: 7,
numberFormat: "###,##0.0##"
}
]
}
}
];
function axisChangeListener(evt) {
var cmd = "fluxtream_date_axis_json = '" + JSON.stringify(evt) + "'";
parent.IPython.notebook.kernel.execute(cmd);
}
window.grapherLoad = function() {
// Request credentials from Python
var callback = function(msg) {
fluxtream_credentials = JSON.parse(eval(msg.content.data['text/plain']));
init();
}
parent.IPython.notebook.kernel.execute(
'json.dumps({"username":fluxtream_username, "password":fluxtream_password, "uid":fluxtream_guest_id})',
{iopub: {output: callback}}, {silent: false});
}
function init() {
dateAxis = new DateAxis("dateAxis", "horizontal", {min: new Date().getTime() / 1000 - 86400, max: new Date().getTime() / 1000});
dateAxis.addAxisChangeListener(axisChangeListener);
for (var i = 0; i < channels.length; i++) {
series[i] = {}
series[i].axis = new NumberAxis('series' + i + 'axis', "vertical", {min: 0.5, max: 1.2});
var datasource = function(level, offset, successCallback, failureCallback) {
$.ajax({
url: 'http://fluxtream-api-proxy.cmucreatelab.org/api/bodytrack/tiles/'
+ fluxtream_credentials.uid + '/' + channels[0].channel + '/'
+ level + '.' + offset + '.json',
success:function(data){successCallback(JSON.stringify(data))},
failure:failureCallback,
headers: {Authorization: 'Basic ' + btoa(fluxtream_credentials.username + ':' + fluxtream_credentials.password)}
});
}
var plot = new DataSeriesPlot(datasource, dateAxis, series[i].axis, {});
plot.setStyle({
styles: channels[i].styles,
highlight: channels[i].highlight
});
series[i].pc = new PlotContainer('series' + i, false, [plot]);
}
$(window).resize(setSizes);
setSizes();
}
function setSizes() {
dateAxis.setSize($('#dateAxis').width(), $("#dateAxis").height(), SequenceNumber.getNext());
for (var i = 0; i < channels.length; i++) {
series[i].axis.setSize($('#series'+i+'axis').width(), $('#series'+i+'axis').height(), SequenceNumber.getNext());
series[i].pc.setSize($('#series'+i).width(), $('#series'+i).height(), SequenceNumber.getNext());
}
}
function displayValue(val) {
$("#valueLabel").html(val ? val['dateString'] + " " + val['valueString'] : "");
}
</script>
</head>
<body>
<div style="width:100%; height:39px">
<div id="dateAxis" style="position: fixed; left:100px; right:30px; height:37px; border:1px solid black"></div>
</div>
<div style="width:100%; height:150px; margin-top:-1px">
<div id="series0title" style="position: fixed; left:0px; width:99px; height:148px; border:1px solid black"></div>
<div id="series0" style="position: fixed; left:100px; right:30px; height:148px; border:1px solid black"></div>
<div id="series0axis" style="position: fixed; right:0px; width:29px; height:148px; border:1px solid black"></div>
</div>
<canvas id="narrow" style="width:100%; height:73px"></canvas>
</body>
</html>
"""
iframe_with_source(src, height='200px')
Out[5]:
In [6]:
start_time = json.loads(fluxtream_date_axis_json)['min']
end_time = json.loads(fluxtream_date_axis_json)['max']
ts_col = []
rr_col = []
for row in fluxtream_get_csv(fluxtream_guest_id, ['PolarStrap.BeatSpacing'], start_time, end_time):
ts_col.append(float(row[0]))
rr_col.append(float(row[1]))
print "Read %d samples from %d to %d into rr_col" % (len(rr_col), start_time, end_time)
cmap = matplotlib.colors.LinearSegmentedColormap.from_list('scaled', [[0,0,0], [0,0,0], [0,0,0], [0,0,0], [0,0,0], [0,0,0], [0,0,0], [0,0,0],
[1,0,0], [1,1,0], [1,1,1]], gamma=1)
plot_hrv(ts_col, rr_col)
In [ ]:
# VLF <0.04 Hz
# LF 0.04 Hz - 0.15 Hz
# HF 0.15 Hz - 0.4 Hz
In [ ]:
# Evaluate this cell to read R to R from local file
# filename = "polar-rr2.csv"
# reader = csv.reader(open(filename,'rb'), delimiter=',')
# skip header
# header = reader.next()