This iPython notebook walks you through reading data from a Netatmo account, either via CSV files downloaded from the Netatmo web site or via use of the Netatmo API, and sending it to Fluxtream.
If you are new to iPython notebooks, here is the main IP[y] website. You will need to install python and iPython notebook on your local system, run a local ipython kernel, and install a local copy of this notebook to be able to execute and modify the code below. Install instructions are here. On osx systems, you can start the server by going to Terminal and calling 'ipython notebook'. This will start a local web server and open a an IP[y] page talking to it in a web browser. Within the IP[y] page, you can open a saved iPython notebooy by going to File/Open.
Once you have IP[y] generally working on your system, here's a brief intro in how to use it:
When a given cell is executed, it may print output which appears below the cell, and the cursor will continue to the next cell. If the next cell is tall, you might need to scroll back up to see the previous cell's output.
Each cell in this notebook pertains to a particular step in the process, topic, or action requiring your input and contains comments at the top saying what it's about and what you should do.
Cells that require entry of sensitive information, such as passwords, start with a phrase like "Execute and fill in the fields below". These generally create entry forms in the output area below the cell that you need to fill in. Generally, the cell will clear the sensitive input boxes after clicking the button. It may also print out suggestions about how you could set up a new cell for future use if you're confident other's won't see your copy of the notebook.
Cells that require customization for your own setup start with "Modify". These include cells where you configure which Netatmo stations and modules you want to upload, what Fluxtream device names you want to associate with each one with, or where you've saved CSV files on your system. These require some thought about which sets of data you want to see in Fluxtream and how you want it to appear.
Cells that define functions or do other things that don't require user input or modification generally just start with "Execute". These can just be executed without much consideration, though you may want to go back later to understand or modify them.
This notebook also includes examples of doing custom processing on the data before sending it to Fluxtream. The functions below which process the Netatmo data read the raw Temperature and Humidity values and calculate a derived metric called H2O_ppm which gives the parts per million of water in the air. They also convert the temperature to Farenheit. Those of you from cultures which foster an intuitive feel for celcius can change this behavior by removing the calls to C2F() in netatmo_csv_to_post_data and/or netatmo_api_to_post_data.
Please enjoy, tinker, modify, etc. Feel free to contact info@fluxtream.org if you have questions.
Note that uploading data multiple times to a given device and channel with identical time values each time will safely overwrite the previous values. However, there is no API or user interaction component in Fluxtream that allows the deletion of a previously-uploaded device or channel, and you can't delete data points already uploaded to a given channel. If you create device names or channel names, or upload data at incorrect timepoints within a given channel, and later regret it, please send the info about your situation, including your Fluxtream username, guest ID, and the details of which devices and or channels you want deleted to info@fluxtream.org. You can get your Guest ID by doing the step below to set up your Fluxtream credentials and looking at the value of fluxtream_guest_id. Also note that the Fluxtream upload API cannot currently handle empty cells within the data array used in an upload call. I'm hoping to fix this in the future.
In [2]:
# Execute to get the h2o_ppm functions to calculate absolute moisture content
# and other nice math for dealing with temperature and humidity.
# Taken from http://a2wh.com/free/humid_dew.py.
""" Python code for calculating dew point and absolute moisture content
from input humidity and temperature
By Joe Ellsworth
(c) Aug 2010 - Free Use for all, No warranty, No promises, No claims
You may use or embed this as desired with no payment to me.
joexdobs@gmail.com
"""
import math
TEMP_K_C_OFFSET = 237.3
UCONST1 = 0.66077
CUBIC_FOOT_PER_CUBIC_METER = 35.3146667
CUBIC_INCH_PER_CUBIC_FOOT = 1728
CUBIC_INCH_PER_CUBIC_YARD = 46656
CUBIC_INCH_PER_GALLON = 230.9851624
CUBIC_FOOT_PER_GALLON = 0.133671969
GALLONS_PER_CUBIC_FOOT = 7.481
GALLONS_PER_CUBIC_METER = 264.17
LITERS_PER_GALLON = 3.7854118
GRAMS_PER_GALLON = 3778.4
GRAMS_PER_POUND = 453.5924
GRAMS_PER_OUNCE = 28.3495231
def C2K(temp_c):
return temp_c + TEMP_K_C_OFFSET
def C2F(temp_c):
return ((temp_c * 9.0)/5.0) + 32.0
def F2C(temp_f):
return ((temp_f - 32.0) * 5.0) / 9.0
def gram2ounce(grams):
return grams / GRAMS_PER_OUNCE
def ounce2gram(ounces):
return ounces * GRAMS_PER_OUNCE
def gram2gallon(grams):
return grams / GRAMS_PER_GALLON
def gallon2gram(gallons):
return gallons * GRAMS_PER_GALLON
def cubicfoot2cubicmeter(feet):
return feet / CUBIC_FOOT_PER_CUBIC_METER
def cubicmeter2cubicfoot(meters):
return meters * CUBIC_FOOT_PER_CUBIC_METER
""" Based in input releative humidity and Temperature
convert a given humidity and temperature in C to a dew point
in C. rel_humid= to the relative humidity as eg 90.5 = 90.5%
temp is temperature C.
Dew Point formula - University of Arizon -
http://cals.arizona.edu/azmet/dewpoint.html
"""
def calc_dew_c(rel_humid, temp_c):
l = math.log(rel_humid / 100.0)
m = 17.27 * temp_c
n = C2K(temp_c)
b = (l + (m / n)) / 17.27
dew_c = (TEMP_K_C_OFFSET * b) / (1 - b)
return dew_c
""" Another formula. This one seems to consistently over state
results by about 0.1C as compared to the previous formula. Neither
formula duplicates the results from our excel formula because the python
log returns slightly different results than the excel ln() funciton. The
difference is normally less than 0.2C.
"""
def calc_dew_c_v1(rel_humid, temp_c):
es0 = 6.11
c1 = 17.27
c2 = 237.7
rhp = rel_humid / 100.0
temp_k = C2K(temp_c)
sat_vapor_pressure_es = es0 * 10 ** (7.5 * temp_c / temp_k)
curr_pressure_e = rhp * sat_vapor_pressure_es
temp_dew_c = (-430.22+237.7* math.log(curr_pressure_e)) / (0 - math.log(curr_pressure_e)+19.08)
#print "rhp=%f sat_vapor_pressure_es=%f curr_pressue_e=%f temp_dew_c=%f" % (
# rhp,sat_vapor_pressure_es,curr_pressure_e, temp_dew_c)
return temp_dew_c
"""Return parts per million of H2O in
air at specified relative humidity
and temp_c"""
def h2o_ppm(rel_humid, temp_c):
CA = 8.1332
CB = 1762.39
CC = 235.66
dew_c = calc_dew_c(rel_humid, temp_c)
curr_temp_part_pressure = CA - (CB / (temp_c + CC))
amb_part_press_mm_hg = 10 ** curr_temp_part_pressure
dew_temp_part_pressure = CA - (CB / (dew_c + CC))
dew_temp_part_press_mm_hg = 10 ** dew_temp_part_pressure
water_ppm = (dew_temp_part_press_mm_hg / 760.0) * 1000000
return water_ppm
def h2o_grams_per_cubic_meter(rel_humid, temp_c):
water_ppm = h2o_ppm(rel_humid, temp_c)
water_gram_per_cubic_meter = water_ppm * 0.001 * 18.0 / 22.4
return water_gram_per_cubic_meter
def h2o_ounce_per_cubic_foot(rel_humid, temp_c):
gpcm = h2o_grams_per_cubic_meter(rel_humid, temp_c)
ounce_per_cubic_meter = gram2ounce(gpcm)
ounce_per_cubic_foot = ounce_per_cubic_meter / CUBIC_FOOT_PER_CUBIC_METER
return ounce_per_cubic_foot
#################
### Functions only to print out results
### for the test harness. Test functions
### where done in F to make easier to enter
### Test.
#################
def test_calc_dew_f(rel_hum,temp_f):
temp_c = F2C(temp_f)
dew_c = calc_dew_c(rel_hum, temp_c)
dew_f = C2F(dew_c)
print "V0 temp C=%5.1F temp F=%5.1F rel_hum=%3.1F%% dew_c=%5.1F dew_f=%5.1F " % (
temp_c, temp_f, rel_hum, dew_c, dew_f)
return dew_f
def test_calc_dew_f_v1(rel_hum,temp_f):
temp_c = F2C(temp_f)
dew_c = calc_dew_c_v1(rel_hum, temp_c)
dew_f = C2F(dew_c)
print "V1 temp C=%5.1F temp F=%5.1F rel_hum=%3.1F%% dew_c=%5.1F dew_f=%5.1F " % (
temp_c, temp_f, rel_hum, dew_c, dew_f)
return dew_f
def test_calc_h2o_ppm(rel_hum,temp_f):
test_calc_dew_f(rel_hum, temp_f)
temp_c = F2C(temp_f)
ppm = h2o_ppm(rel_hum, temp_f)
h2ogm = h2o_grams_per_cubic_meter(rel_hum, temp_c)
h2oopf3 = h2o_ounce_per_cubic_foot(rel_hum, temp_c)
print "V0 temp ppm=%7.0F gram cubic meter=%5.1f Once cubic Foot=%7.4F" % (
ppm,h2ogm,h2oopf3)
return ppm
# - - - - - - - - - - - - - - -
def test_and_sample_use():
# - - - - - - - - - - - - - - -
print "100C as F=", C2F(100)
print "212F as C=", F2C(212)
print "32F as C=", F2C(32)
dew_c = calc_dew_c(90, 36)
print "dew c = ", dew_c
test_calc_dew_f(rel_hum=80.0, temp_f=100.0)
test_calc_dew_f(rel_hum=20.0, temp_f=80.0)
test_calc_dew_f(rel_hum=40.0, temp_f=80.0)
test_calc_dew_f(rel_hum=80.0, temp_f=80.0)
test_calc_dew_f_v1(rel_hum=80.0, temp_f=100.0)
test_calc_dew_f_v1(rel_hum=20.0, temp_f=80.0)
test_calc_dew_f_v1(rel_hum=40.0, temp_f=80.0)
test_calc_dew_f_v1(rel_hum=80.0, temp_f=80.0)
ppm = test_calc_h2o_ppm(80.0,80) #Answer=21.969
ppm = test_calc_h2o_ppm(40.0,80) #Answer=10.950
ppm = test_calc_h2o_ppm(20.0,80)
ppm = test_calc_h2o_ppm(10.0,80)
h2ogpm3 = h2o_grams_per_cubic_meter(40,26.6667)
h2oopf3 = h2o_ounce_per_cubic_foot(40, 26.667)
In [3]:
# Execute this cell to define the functions for calling the Fluxtream upload API for the
# credentials entered below
import json, subprocess
import datetime
from dateutil import tz
def epoch_time(dt):
epoch = datetime.datetime(1970, 1, 1, tzinfo=tz.tzutc())
return (dt - epoch).total_seconds()
# By default, the upload function will send data to the main server at fluxtream.org.
# If you want to have this use a different fluxtream server, change it here
# and make sure the username and password entered below are valid on that server.
global fluxtream_server
fluxtream_server = "fluxtream.org"
def setup_fluxtream_credentials():
# Call the Fluxtream guest API, documented at
# https://fluxtream.atlassian.net/wiki/display/FLX/BodyTrack+server+APIs#BodyTrackserverAPIs-GettheIDfortheguest
# Make sure it works and harvest the Guest ID for future use
global fluxtream_server, fluxtream_username, fluxtream_password, fluxtream_guest_id
# Make sure we have fluxtream credentials set properly
if not('fluxtream_server' in globals() and
'fluxtream_username' in globals() and
'fluxtream_password' in globals()):
raise Exception("Need to enter Fluxtream credentials before uploading data. See above.")
cmd = ['curl', '-v']
cmd += ['-u', '%s:%s' % (fluxtream_username, fluxtream_password)]
cmd += ['https://%s/api/guest' % fluxtream_server]
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
try:
response = json.loads(result_str)
if 'id' in response:
fluxtream_guest_id = int(response['id'])
else:
raise Exception('Received unexpected response %s while trying to check credentials for %s on %s' % (response,
fluxtream_username,
fluxtream_server))
print 'Verified credentials for user %s on %s work. Guest ID=%d' % (fluxtream_username, fluxtream_server, fluxtream_guest_id)
except:
print "Attempt to check credentials of user %s failed" % (fluxtream_username)
print "Server returned response of: %s" % (result_str)
print "Check login to https://%s works and re-enter your Fluxtream credentials above" % (fluxtream_server)
raise
def fluxtream_upload(dev_nickname, channel_names, data):
global fluxtream_server, fluxtream_username, fluxtream_password
# Make sure we have some data to send
if data == None or len(data)<1:
print 'Nothing to upload to %s %s' % (dev_nickname, channel_names)
return
# Make sure we have fluxtream credentials set properly
if not('fluxtream_server' in globals() and
'fluxtream_username' in globals() and
'fluxtream_password' in globals()):
raise Exception("Need to enter Fluxtream credentials before uploading data. See above.")
# Send to BodyTrack upload API, documented at
# https://fluxtream.atlassian.net/wiki/display/FLX/BodyTrack+server+APIs#BodyTrackserverAPIs-Storingdata
cmd = ['curl', '-v']
cmd += ['-u', '%s:%s' % (fluxtream_username, fluxtream_password)]
cmd += ['-d', 'dev_nickname=%s' % dev_nickname]
cmd += ['-d', 'channel_names=%s' % json.dumps(channel_names)]
cmd += ['-d', 'data=%s' % json.dumps(data)]
cmd += ['https://%s/api/bodytrack/upload' % fluxtream_server]
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
try:
response = json.loads(result_str)
if response['result'] != 'OK':
raise Exception('Received non-OK response %s while trying to upload to %s' % (response, dev_nickname))
print 'Upload to %s %s (%d rows, %d to %d) succeeded' % (dev_nickname, channel_names, len(data), data[0][0], data[-1][0])
except:
print "Attempt to upload to %s as user %s failed. Check that your credentials are ok" % (fluxtream_server,
fluxtream_username)
print "Server returned response: %s" % (result_str)
raise
In [15]:
# Execute and fill in the fields below to set your Fluxtream credentials.
from IPython.html import widgets # Widget definitions
from IPython.display import display # Used to display widgets in the notebook
def set_fluxtream_password(this):
global fluxtream_username, fluxtream_password
fluxtream_username = fluxtream_username_widget.value
fluxtream_password = fluxtream_password_widget.value
fluxtream_password_widget.value = ''
setup_fluxtream_credentials()
print "To make persistent for future restarts, insert a cell, paste in:"
print ""
print "global fluxtream_username, fluxtream_password"
print "fluxtream_username = \"%s\"" % (fluxtream_username)
print "fluxtream_password = \"xxx\""
print "setup_fluxtream_credentials()"
print ""
print "replace xxx with your password, and execute that cell instead."
print "Only do this if you're keeping this copy of your iPython notebook private,"
print "and remove that cell before sharing"
display(widgets.HTMLWidget(value='Fluxtream Username'))
fluxtream_username_widget = widgets.TextWidget()
display(fluxtream_username_widget)
display(widgets.HTMLWidget(value='Fluxtream Password'))
fluxtream_password_widget = widgets.TextWidget()
display(fluxtream_password_widget)
set_fluxtream_login_button = widgets.ButtonWidget(description='Set Fluxtream credentials')
set_fluxtream_login_button.on_click(set_fluxtream_password)
display(set_fluxtream_login_button)
# Enter Fluxtream username and password and click "Set Fluxtream credentials" button.
# Password field will blank afterwards, but variables will be set
In [5]:
# Execute this cell to define the functions for reading in Netatmo CSV files.
# Skip to the next section if you want to use the API method instead.
# You can download CSV by going to
# http://my.netatmo.com/app/station, logging in, clicking on
# the icon along the right next to the station name, just below the main icon bar
# (looks like a pair of Netatmo devices next to a gear), scrolling down to
# click on Advanced > Download your station data as a CSV/XLS file,
# selecting the module, changing file format to csv, setting date/time range,
# and clicking the Download button.
# Keep track of where you save the files, and enter in the next section
import httplib, urllib, time, base64, string, datetime, json, csv, calendar
from dateutil import tz
from dateutil import parser
# Returns array of channel names produced by function below
def netatmo_csv_channel_names():
#return ["Temp_F","RH","H2O_ppm","CO2"]
return ["Temp_F","RH","H2O_ppm"]
# Returns 2D array of data suitable for posting to Fluxtream API
def netatmo_csv_to_post_data(filename):
reader = csv.reader(open(filename,'rb'), delimiter=';')
# skip 4 rows of headers
for i in range(0,4):
header = reader.next()
# Fourth line of header includes timezone as second item. Example:
# Timestamp;"Timezone : America/New_York";Temperature;Humidity;CO2
# Parse out the timezone by splitting Timezone string on spaces and taking the last element
tz_str = header[1].split(" ")[-1]
local_tz = tz.gettz(tz_str)
#print "Using timezone %s (%s)" % (tz_str, local_tz)
#Format of body of netatmo 'csv' file:
# Unixtime;Local time;Temperature;Humidity;CO2
# Units of Temperature are Celcius
# Convert to Farenheit for data array
rowcount = 0;
data = []
for row in reader:
unix_ts = int(row[0])
temp_c = float(row[2])
rh = int(row[3])
ppm = h2o_ppm(rh, temp_c)
# co2 = int(row[4])
# data.append([unix_ts, C2F(temp_c), rh, ppm, co2])
data.append([unix_ts, C2F(temp_c), rh, ppm])
# local_ts = datetime.datetime.strptime(row[1], '%Y/%m/%d %H:%M:%S')
# local_ts = local_ts.replace(tzinfo=local_tz)
# print "%d: %d, %s (%d)" % (rowcount, unix_ts, local_ts, epoch_time(local_ts))
# rowcount+=1
return data
In [6]:
# Modify the values below for uploading Netatmo CSV data to your Fluxtream account.
# Skip if you want to use the API method below
# These examples are what we used for Anne and Randy's home Netatmo setup and
# where Anne downloaded the CSV files onto her laptop
# Change the arg to netatmo_csv_to_post_data to be where you saved your CSV files
# Change the first arg to fluxtream_upload the the device names you want to see the data
# channels under within the Fluxtream BodyTrack app
# Then execute the cell
basement_data = netatmo_csv_to_post_data("/Users/anne/home/pgh-info/pgh-apt/Basement_18-5-2014-v2.csv")
porch_data = netatmo_csv_to_post_data("/Users/anne/home/pgh-info/pgh-apt/Porch_18-5-2014-v2.csv")
fluxtream_upload("Netatmo_Basement", netatmo_csv_channel_names(), basement_data)
fluxtream_upload("Netatmo_Porch", netatmo_csv_channel_names(), porch_data)
In [8]:
# Execute and fill in the fields below to set your Netatmo API developer credentials.
# Before doing this for the first time, you need to apply for a client id and secret
# by logging into Netatmo and creating an application at
# https://dev.netatmo.com/dev/createapp
# Once you fill out the form and click the CREATE button, you'll see a page with
# lines that look like:
# Client id xxxxxxxxxxxxxxxxxxxxxxxx
# Client secret xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Keep these values safe. Each time you restart the iPython kernel,
# execute this cell and enter your developer credentials in the form below,
# or follow the instructions it prints to create a new cell for your private use.
from IPython.html import widgets # Widget definitions
from IPython.display import display # Used to display widgets in the notebook
def set_netatmo_dev_credentials(this):
global netatmo_client_id, netatmo_client_secret
netatmo_client_id = netatmo_client_id_widget.value.strip('" ')
netatmo_client_secret = netatmo_client_secret_widget.value.strip('" ')
netatmo_client_secret_widget.value = ''
print "Netatmo API credentials set successfully"
print "To make persistent for future restarts, insert a cell, paste in:"
print ""
print "global netatmo_client_id, netatmo_client_secret"
print "netatmo_client_id = \"%s\"" % (netatmo_client_id)
print "netatmo_client_secret = \"xxx\""
print ""
print "replace xxx with your client_secret, and execute that cell instead."
print "Only do this if you're keeping this copy of your iPython notebook private,"
print "and remove that cell before sharing"
display(widgets.HTMLWidget(value='Netatmo Client id'))
netatmo_client_id_widget = widgets.TextWidget()
display(netatmo_client_id_widget)
display(widgets.HTMLWidget(value='Netatmo Client secret'))
netatmo_client_secret_widget = widgets.TextWidget()
display(netatmo_client_secret_widget)
set_netatmo_dev_credentials_button = widgets.ButtonWidget(description='Set Netatmo API credentials')
set_netatmo_dev_credentials_button.on_click(set_netatmo_dev_credentials)
display(set_netatmo_dev_credentials_button)
# Enter Netatmo API credentials and click button below.
# Client secret field will blank afterwards, but variables will be set
In [6]:
# Execute this cell to define the functions for supporting OAuth access token
# handling
import time, datetime, subprocess, json
def authorize_netatmo_api_access(netatmo_username, netatmo_password):
# This function uses the "Client Credentials" method to authorize OAuth2 access to Netatmo.
# See https://dev.netatmo.com/doc/authentication/usercred for details
# This only needs to be done once for binding a given user's Netatmo account to your client_id.
# Save the resulting netatmo_refresh_token in a safe place with your client_id and client_secret
# for future runs and/or restarts of the iPython kernel.
global netatmo_client_id, netatmo_client_secret, netatmo_refresh_token, netatmo_access_token, netatmo_access_exp
if not('netatmo_client_id' in globals() and 'netatmo_client_secret' in globals()):
raise Exception("Need to enter netatmo_client_id and netatmo_client_secret before refreshing access token. See above.")
cmd = ['curl', '-v']
cmd += ['-d', 'grant_type=password']
cmd += ['-d', 'client_id=%s' % netatmo_client_id]
cmd += ['-d', 'client_secret=%s' % netatmo_client_secret]
cmd += ['-d', 'username=%s' % netatmo_username]
cmd += ['-d', 'password=%s' % netatmo_password]
cmd += ['-d', 'scope=read_station']
cmd += ['https://api.netatmo.net/oauth2/token']
# print "Executing %s" % cmd
refresh_req_time = time.time()
result_str = subprocess.check_output(cmd)
# print ' Result=%s' % (result_str)
response = json.loads(result_str)
if not('refresh_token' in response and 'access_token' in response and 'expires_in' in response):
raise Exception('Received unexpected response %s while trying to authorize Netatmo API tokens' % (response))
netatmo_refresh_token = response['refresh_token']
netatmo_access_token = response['access_token']
netatmo_access_exp = refresh_req_time + int(response['expires_in'])
print "Successfully authorized netatmo_access_token, expires %s (%d)" % (datetime.datetime.fromtimestamp(netatmo_access_exp),
netatmo_access_exp)
print "To make persistent for future restarts, insert a cell, paste in:"
print ""
print " netatmo_refresh_token = \"%s\"" % (netatmo_refresh_token)
print " refresh_netatmo_access_token()"
print ""
print "and execute that cell instead"
def refresh_netatmo_access_token():
# This function uses an existing refresh_token, along with the matching client_id and client_secret
# to get a fresh access_token to allow API access to data in a given Netatmo user's account
# See https://dev.netatmo.com/doc/authentication/refreshtoken for details
# Call this when you first set netatmo_refresh_token after a given iPython kernal restart or when
# an existing access_token expires (time.time()>netatmo_access_exp)
global netatmo_client_id, netatmo_client_secret, netatmo_refresh_token, netatmo_access_token, netatmo_access_exp
if not('netatmo_client_id' in globals() and 'netatmo_client_secret' in globals()):
raise Exception("Need to enter netatmo_client_id and netatmo_client_secret before refreshing access token. See above.")
if not('netatmo_refresh_token' in globals()):
raise Exception("Need to enter netatmo_refresh_token before refreshing access token. See above.")
cmd = ['curl', '-v']
cmd += ['-d', 'grant_type=refresh_token']
cmd += ['-d', 'refresh_token=%s' % netatmo_refresh_token]
cmd += ['-d', 'client_id=%s' % netatmo_client_id]
cmd += ['-d', 'client_secret=%s' % netatmo_client_secret]
cmd += ['https://api.netatmo.net/oauth2/token']
refresh_req_time = time.time()
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
response = json.loads(result_str)
if not('access_token' in response and 'expires_in' in response):
raise Exception('Received unexpected response %s while trying to refresh Netatmo API tokens' % (response))
netatmo_access_token = response['access_token']
netatmo_access_exp = refresh_req_time + int(response['expires_in'])
print "Successfully refreshed netatmo_access_token, expires %s (%d)" % (datetime.datetime.fromtimestamp(netatmo_access_exp),
netatmo_access_exp)
def check_netatmo_access_token():
# Make sure either netatmo_access_token and netatmo_access_exp are set and non-expired,
# or try to refresh the Netatmo access_token
global netatmo_access_token, netatmo_access_exp
if ('netatmo_access_token' in globals() and 'netatmo_access_exp' in globals() and time.time()<netatmo_access_exp):
# Looks ok
return
# Either the access_token isn't set or it's expired. Try to refresh it
refresh_netatmo_access_token()
In [10]:
# Use this cell ONLY when you need to do the initial dance to grant
# OAuth access to a given user's netatmo account. You only need to do this
# once for binding the account for a given user to your Netatmo client_id.
# Hold onto the refresh_token it returns for future use (see below).
# Note that the Netatmo account that the API developer credentials (client_id, client_secret)
# are registered with doesn't need to be the same as the Netatmo account
# that you're binding to.
from IPython.html import widgets # Widget definitions
from IPython.display import display # Used to display widgets in the notebook
def bind_netatmo_account(this):
global netatmo_refresh_token, netatmo_access_token, netatmo_access_exp
# Harvest the user's Netatmo login credentials from the form
netatmo_username = netatmo_username_widget.value
netatmo_password = netatmo_password_widget.value
authorize_netatmo_api_access(netatmo_username, netatmo_password)
# Clear the user's Netatmo login credentials from the form and python variables
netatmo_username_widget.value = ''
netatmo_password_widget.value = ''
netatmo_username = None
netatmo_password = None
display(widgets.HTMLWidget(value='<b>Enter credentials for the Netatmo account you want to get data from</b>'))
display(widgets.HTMLWidget(value='Netatmo Username'))
netatmo_username_widget = widgets.TextWidget()
display(netatmo_username_widget)
display(widgets.HTMLWidget(value='Netatmo Password'))
netatmo_password_widget = widgets.TextWidget()
display(netatmo_password_widget)
set_netatmo_bind_button = widgets.ButtonWidget(description='Bind to this Netatmo User Account')
set_netatmo_bind_button.on_click(bind_netatmo_account)
display(set_netatmo_bind_button)
In [6]:
# Use this cell ONLY when you restart the iPython kernel after having done
# the initial dance to grant OAuth access to a given user's netatmo account
# and chose not to create a cell in this notebook to make the refresh_token
# setting persistent
from IPython.html import widgets # Widget definitions
from IPython.display import display # Used to display widgets in the notebook
# Call this when you first set netatmo_refresh_token or when
# an existing access_token expires (time.time()>netatmo_access_exp)
def set_netatmo_refresh_token(this):
global netatmo_refresh_token, netatmo_access_token, netatmo_access_exp
netatmo_refresh_token = netatmo_refresh_token_widget.value.strip('" ')
refresh_netatmo_access_token()
print "To make persistent for future restarts, insert a cell, paste in:"
print ""
print "global netatmo_refresh_token, netatmo_access_token, netatmo_access_exp"
print "netatmo_refresh_token = \"%s\"" % (netatmo_refresh_token)
print "refresh_netatmo_access_token()"
print ""
print "and execute that cell instead"
display(widgets.HTMLWidget(value='Netatmo Refresh token'))
netatmo_refresh_token_widget = widgets.TextWidget()
display(netatmo_refresh_token_widget)
set_netatmo_refresh_token_button = widgets.ButtonWidget(description='Set Netatmo Refresh token')
set_netatmo_refresh_token_button.on_click(set_netatmo_refresh_token)
display(set_netatmo_refresh_token_button)
In [11]:
# Execute this cell to define functions to support calling the Netatmo API
# Assumes that the Netatmo API dev credentials are set up already,
# Either netatmo_access_token and netatmo_access_exp must be valid and non-expired,
# or netatmo_client_id, netatmo_client_secret, and netatmo_refresh_token must be valid
import datetime, json, os, pprint, subprocess
def netatmo_get_devlist():
check_netatmo_access_token()
cmd = ['curl', '-v']
cmd += ['http://api.netatmo.net/api/devicelist?access_token=%s' % netatmo_access_token]
result_str = subprocess.check_output(cmd)
#print ' Result=%s' % (result_str)
response = json.loads(result_str)
return response
def netatmo_get_station_names(devlist):
station_names = []
for dev in devlist['body']['devices']:
station_names.append(dev['station_name'])
return station_names
def netatmo_get_station_info(station_name, devlist):
for dev in devlist['body']['devices']:
if(dev['station_name'] == station_name):
return dev
return None
def netatmo_get_module_names(station_name, devlist):
station_dev = netatmo_get_station_info(station_name, devlist)
station_id = station_dev['_id']
# A station itself is also a module, so we need to include that, plus
# any additional modules with this station as their main_device
module_names = [ station_dev['module_name'] ]
for module in devlist['body']['modules']:
if module['main_device'] == station_id:
module_names.append(module['module_name'])
return module_names
def netatmo_get_module_info(station_name, module_name, devlist):
station_dev = netatmo_get_station_info(station_name, devlist)
station_id = station_dev['_id']
# A station itself is also a module, so we need to check if
# the module_name is for the station itself, then check the
# modules array for other possibilities
if station_dev['module_name'] == module_name:
# Modify station_dev to contain 'main_device' field with its own id
# so we can treat it consistently with other modules
station_dev['main_device'] = station_id
return station_dev
for module in devlist['body']['modules']:
if module['main_device'] == station_id and module['module_name'] == module_name:
return module
return None
def netatmo_get_module_data(module_info, start_ts, end_ts):
check_netatmo_access_token()
cmd = ['curl', '-v']
cmd += ['http://api.netatmo.net/api/getmeasure?access_token=%s&device_id=%s&module_id=%s&scale=max&type="Temperature,Humidity,CO2"&date_begin=%d&date_end=%d&optimize=false' %
(netatmo_access_token, module_info['main_device'], module_info['_id'], start_ts, end_ts)]
# print ' Executing %s' % (cmd)
result_str = subprocess.check_output(cmd)
# print ' Result=%s' % (result_str)
response = json.loads(result_str)
if response['status'] != 'ok':
raise Exception('Received non-OK response while trying to query data for %s: %s' % (module_info['module_name'], response))
return response
# Returns array of channel names produced by function below
def netatmo_api_channel_names():
return ["Temp_F","RH","H2O_ppm"]
#return ["Temp_F","RH","H2O_ppm","CO2"]
# Returns 2D array of data suitable for posting to Fluxtream API
# The input data for this is the parsed json returned by netatmo_get_module_data
def netatmo_api_to_post_data(module_data_response):
# Units of Temperature are Celcius
# Convert to Farenheit for data array
rowcount = 0;
data = []
for unix_ts_str in sorted(module_data_response['body'].keys()):
vals = module_data_response['body'][unix_ts_str]
unix_ts = int(unix_ts_str)
temp_c = float(vals[0])
rh = int(vals[1])
ppm = h2o_ppm(rh, temp_c)
#co2 = int(vals[2])
#data.append([unix_ts, C2F(temp_c), rh, ppm, co2])
data.append([unix_ts, C2F(temp_c), rh, ppm])
#print "%s" % (data[-1])
# rowcount+=1
return data
In [12]:
# Execute this once after iPython kernel start and/or change of the Netatmo user the access_token is bound to
# This executes a Netatmo API call to set up devlist with the Netatmo user's station and module configuration.
devlist = netatmo_get_devlist()
print "Successfully setup devlist"
# Iterate over the stations and print the modules associated with each. This isn't strictly needed, but is helpful
# since all the APIs need the station and module names to navigate the devlist
station_names = netatmo_get_station_names(devlist)
for station_name in station_names:
mod_names = netatmo_get_module_names(station_name, devlist)
print "Station '%s', %d modules:" % (station_name, len(mod_names))
print " %s" % (mod_names)
In [13]:
# Modify the values below for setting up how which Netatmo data to send to your Fluxtream account
# and which device names you want to see the data channels under within the Fluxtream BodyTrack app
# These examples are what we used for Anne and Randy's home Netatmo setup
# Change the keys of devname_map to the Fluxtream device names you want to use.
# Change the values in devname_map to strings consisting of Netatmo's <station_name>/<module_name>
# Execute to setup module_info_map based on those settings.
# The output of the cell above shows what the station and modules names are for the
# Netatmo account you've bound the access_token to.
devname_map = {'Netatmo_Basement': 'AR Home/Basement',
'Netatmo_Porch': 'AR Home/Porch',
'Netatmo_Bedroom': 'AR Home/Bedroom',
'Netatmo_Office': 'AR Home/Office'
}
# Parse devlist to populate module_info_map to map from Fluxtream device names to Netatmo module specifications
module_info_map = {}
for devname in devname_map.keys():
# Split the station and module names
netatmo_name_elts = devname_map[devname].split('/')
# Get the module_info object for this
module_info = netatmo_get_module_info(netatmo_name_elts[0],netatmo_name_elts[1], devlist)
if module_info == None:
raise Exception("Can't find module info for %s; recheck station and module names list" % devname_map[devname])
# Store the module info in module_info_map
module_info_map[devname]=module_info
print "Successfully setup module_info_map"
In [210]:
# This sample time bounds setting block gets data from a particular date range
start_ts = epoch_time(datetime.datetime.strptime('2013-01-01 00:00:00' , '%Y-%m-%d %H:%M:%S').replace(tzinfo=tz.tzlocal()))
end_ts = epoch_time(datetime.datetime.strptime('2014-01-01 00:00:00' , '%Y-%m-%d %H:%M:%S').replace(tzinfo=tz.tzlocal()))
In [16]:
# This sample time bounds setting block gets data from the last 3 hours.
end_ts = time.time()
start_ts = end_ts - 10800
In [14]:
# This sample time bounds setting block gets data from a set start time until now.
start_ts = epoch_time(datetime.datetime.strptime('2014-05-18 00:00:00' , '%Y-%m-%d %H:%M:%S').replace(tzinfo=tz.tzlocal()))
end_ts = time.time()
In [41]:
# This sample time bounds setting block gets another incremental block starting from the end of the last upload
# You have to use some other method to come up with start_ts if this is the first run or if the kernel is restarted
global last_end_ts
start_ts = last_end_ts
end_ts = time.time()
In [17]:
# Execute to read data over the Netatmo API and upload to Fluxtream
# Make sure to execute a blocks like the ones above to set start_ts and end_ts before each run
# Initialize last_end_ts to be as late as we ask for. May be reduced if the
# minimum of the latest timestamps currently available are smaller.
last_end_ts = end_ts
# Hold onto latest data for each module
latest_data_map = {}
print "Processing %d to %d for %s:" % (start_ts, end_ts, module_info_map.keys())
# Iterate over each of the modules and send data to Fluxtream
# For each one, we might need to loop multiple times since netatmo returns
# a maximum of 1024 values
for devname in module_info_map.keys():
loop_start_ts = start_ts
while True:
api_response = netatmo_get_module_data(module_info_map[devname], loop_start_ts, end_ts)
api_post_data = netatmo_api_to_post_data(api_response)
# Store the latest data for later printing
if(len(api_post_data)>0):
latest_data = api_post_data[-1]
latest_data_map[devname]=latest_data
else:
# Didn't get any data points, assume we're done
break
# Upload the data to Fluxtream
fluxtream_upload(devname, netatmo_api_channel_names(), api_post_data)
# Check if we're done. Assume that if we have exactly 1024 values we should try again
if(len(api_post_data)<1024):
# Done, break out of the loop
break
else:
# Not done yet, loop again with a new loop_start_ts just after the end of the last data we got
loop_start_ts = latest_data[0]+1
# Record the min of the last timestamp and last_end_ts for incremental use so we don't miss data next time.
last_end_ts = min(last_end_ts, latest_data[0])
print "Latest data: %s (%d)" % (datetime.datetime.fromtimestamp(last_end_ts), last_end_ts)
for devname in module_info_map.keys():
print " %20s: %s" % (devname, latest_data_map[devname])
In [ ]: