Time how long it takes to get data via ERDDAP.
Here we are using the IOOS NERACOOS ERDDAP service at http://www.neracoos.org/erddap.
Lots of factors affect timing, as described at the end.
But for these tests it's pretty darn speedy!
In [1]:
%matplotlib inline
import urllib, json
import pandas as pd
Try CSV response for 1 week of accelerometer (wave) data from NERACOOS Buoy B off Portland, ME
In [2]:
url = 'http://www.neracoos.org/erddap/tabledap/B01_accelerometer_all.csv?time,significant_wave_height&time>"now-7days"'
print(url)
In [3]:
%%timeit
df_sb = pd.read_csv(url,index_col='time',parse_dates=True,skiprows=[1]) # skip the units row
In [4]:
df_sb = pd.read_csv(url,index_col='time',parse_dates=True,skiprows=[1]) # skip the units row
df_sb.plot(figsize=(12,4),grid='on');
How about 1 year of data?
In [18]:
url = 'http://www.neracoos.org/erddap/tabledap/B01_accelerometer_all.csv?time,significant_wave_height&time>"now-365days"'
print(url)
In [19]:
%%timeit
df_sb = pd.read_csv(url,index_col='time',parse_dates=True,skiprows=[1]) # skip the units row
In [20]:
df_sb = pd.read_csv(url,index_col='time',parse_dates=True,skiprows=[1]) # skip the units row
df_sb.plot(figsize=(12,4),grid='on');
Try JSON response -- might want to use this in a web app
In [8]:
url = 'http://www.neracoos.org/erddap/tabledap/B01_accelerometer_all.json?time,significant_wave_height&time>"now-7days"'
print(url)
In [9]:
%%timeit
response = urllib.urlopen(url)
data = json.loads(response.read())
In [10]:
response = urllib.urlopen(url)
data = json.loads(response.read())
In [11]:
data
Out[11]:
In general the time to retrieve data depends on:
Bob Simons from NOAA adds: "Some timing issues are counter-intuitive. For example, .nc files are binary so you would expect a request for a .nc file to be done very quickly. But .nc files can't be streamed. So they must be completely created on the server, then transmitted. In contrast, a .csv file can be streamed (ERDDAP reads from a file, writes a chunk to the user, reads another file, writes another chunk to the user). And although csv files are verbose, compression during transmission means that the number of bytes transmitted may actually be less than an uncompressed .nc file. So the actual results are hard to predict -- there are lots of factors."
But this is encouraging, as this suggests ERDDAP would be fast enough to acts as a backend data system to build a dashboard to status a bunch of instruments