API Time Selectors - Using Start, End, and Reftime with Forecast Data

Forecast datasets typically have two time dimensions, reftime and time. The reftime of a forecast data product represents the run time of the forecast model, while time represents the actual time (typically in the future) that the values are valid. For each reftime, values are unique for each unique time value.

Typically the most recent forecast is most desirable to users, and the Planet OS API intentionally returns these values by default. However, there may be cases when one wants to investigate a historical forecast, or how values at a specific time differ between successive forecasts.

The RTOFS data product is a daily updated forecast that provides a 3-day, hourly forecast. This notebook highlights ways to use the start, end, reftime_start, and reftime_end API parameters and explain the expected response.


In [1]:
%matplotlib inline

import pandas as pd
import simplejson as json
from urllib.parse import urlencode
from urllib.request import urlopen, Request
import datetime
import numpy as np

A Planet OS API key is required to run this notebook. Keys are displayed in the account settings page on the Planet OS Datahub. If you do not have a Planet OS account, you can sign up for free.


In [2]:
apikey = open('APIKEY').readlines()[0].strip() #'<YOUR API KEY HERE>'

Request the Most Recent Model Run

By default, our API returns the most recent model run. The example below returns all variables for the most recent forecast run (e.g. reftime). At the time of running, this was 2016-11-12T00:00:00. Given the hourly resolution and 3-day extent, we expect 72 unique values to be returned.


In [3]:
# Set the Planet OS API query parameters
count = 100
id = 'noaa_rtofs_surface_1h_diag'
lat=39.37858604638528
lon=-72.57739563685647
time_order = 'desc'

query_dict = {'apikey': apikey,
              'count': count,
              'lon': lon,
              'lat': lat,
              'time_order': time_order,
              }
query = urlencode(query_dict)

api_url = "http://api.planetos.com/v1/datasets/%s/point?%s" % (id, query)

request = Request(api_url)
response = urlopen(request)
response_json = json.loads(response.read())

data = response_json['entries']

# let's flatten the response and create a Pandas dataframe
df = pd.io.json.json_normalize(data)

# then index by time using the axes.time column
pd.to_datetime(df["axes.time"])
df.set_index('axes.time', inplace=False)

print(df.count())
df.head()


axes.latitude                            73
axes.longitude                           73
axes.reftime                             73
axes.time                                73
classifiers.reference_time               73
context                                  73
data.ice_coverage                        73
data.ice_thickness                       73
data.mixed_layer_thickness               73
data.ssh                                 73
data.surface_boundary_layer_thickness    73
data.u_barotropic_velocity               73
data.v_barotropic_velocity               73
dtype: int64
Out[3]:
axes.latitude axes.longitude axes.reftime axes.time classifiers.reference_time context data.ice_coverage data.ice_thickness data.mixed_layer_thickness data.ssh data.surface_boundary_layer_thickness data.u_barotropic_velocity data.v_barotropic_velocity
0 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-18T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.393272 -0.586881 1.003857 0.052923 -0.051910
1 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T23:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.461914 -0.582981 1.005212 0.048648 -0.042568
2 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T22:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.939150 -0.577908 1.008560 0.047611 -0.033235
3 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T21:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.904634 -0.573326 1.012362 0.049494 -0.023227
4 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T20:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.972110 -0.571742 1.015378 0.052853 -0.012663

Request a Single Model Run

By default, the system will return the most recent model run (e.g. reftime). However, a specific model run can be acquired by requesting an identical reftime_start and reftime_end value. In the example below, we request values from only the 2016-11-12T00:00:00 model run.


In [4]:
df


Out[4]:
axes.latitude axes.longitude axes.reftime axes.time classifiers.reference_time context data.ice_coverage data.ice_thickness data.mixed_layer_thickness data.ssh data.surface_boundary_layer_thickness data.u_barotropic_velocity data.v_barotropic_velocity
0 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-18T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.393272 -0.586881 1.003857 0.052923 -0.051910
1 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T23:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.461914 -0.582981 1.005212 0.048648 -0.042568
2 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T22:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.939150 -0.577908 1.008560 0.047611 -0.033235
3 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T21:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.904634 -0.573326 1.012362 0.049494 -0.023227
4 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T20:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.972110 -0.571742 1.015378 0.052853 -0.012663
5 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T19:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.099548 -0.573966 1.016540 0.056213 -0.002916
6 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T18:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.365773 -0.577964 1.015528 0.059150 0.003989
7 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T17:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.068107 -0.581208 1.012883 0.062220 0.006222
8 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T16:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 4.017891 -0.582298 1.009745 0.066123 0.003401
9 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T15:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 5.489046 -0.580531 1.007133 0.071520 -0.003411
10 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T14:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.375851 -0.576848 1.005445 0.077729 -0.012819
11 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T13:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.100214 -0.572208 1.779485 0.083688 -0.023142
12 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T12:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.878569 -0.566653 4.922035 0.087533 -0.033732
13 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T11:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 8.117210 -0.562986 7.326427 0.088285 -0.044048
14 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T10:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 8.067168 -0.563954 8.341889 0.085997 -0.053084
15 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T09:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.911420 -0.567991 8.159277 0.081106 -0.059898
16 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T08:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.622293 -0.573117 7.675351 0.075394 -0.062901
17 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T07:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.980652 -0.578922 6.264008 0.069813 -0.061442
18 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T06:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 5.889813 -0.583953 4.226895 0.064880 -0.055497
19 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T05:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 4.677490 -0.587404 2.576871 0.061353 -0.045680
20 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T04:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 3.624086 -0.587819 1.619146 0.058696 -0.035362
21 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T03:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 3.030861 -0.583119 2.229293 0.057034 -0.027551
22 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T02:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.669391 -0.575806 2.036594 0.057097 -0.024253
23 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T01:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.870689 -0.566830 1.006377 0.059213 -0.025850
24 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-17T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.931675 -0.559775 1.007717 0.063999 -0.030282
25 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T23:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.785911 -0.556264 1.009856 0.070086 -0.035647
26 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T22:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.789303 -0.556530 1.012161 0.076357 -0.040276
27 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T21:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.808828 -0.561607 1.013881 0.081628 -0.044166
28 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T20:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.840858 -0.570146 1.014458 0.084696 -0.048383
29 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T19:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 0.891038 -0.578862 1.013723 0.086110 -0.052792
... ... ... ... ... ... ... ... ... ... ... ... ... ...
43 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T05:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.560007 -0.559192 1.010422 0.101492 -0.075716
44 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T04:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.883909 -0.559145 1.008574 0.109294 -0.084072
45 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T03:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.600063 -0.556311 1.007095 0.115196 -0.096286
46 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T02:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.387641 -0.553329 1.006572 0.117911 -0.109475
47 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T01:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.241458 -0.551900 1.007071 0.116294 -0.120298
48 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-16T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.296358 -0.551413 1.008421 0.110358 -0.126584
49 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T23:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.471651 -0.552021 1.010378 0.101472 -0.127311
50 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T22:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 1.829469 -0.553539 1.012423 0.090876 -0.123713
51 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T21:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 2.514715 -0.556953 1.013952 0.081366 -0.116807
52 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T20:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 3.332845 -0.561778 1.014560 0.074210 -0.108862
53 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T19:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 3.905654 -0.563903 1.013991 0.070646 -0.100568
54 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T18:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 4.684608 -0.565803 1.012341 0.071928 -0.092220
55 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T17:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.014578 -0.566297 1.010074 0.076021 -0.085030
56 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T16:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.824544 -0.565571 1.007725 0.081834 -0.078454
57 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T15:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.435776 -0.563867 1.005924 0.086961 -0.073020
58 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T14:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 8.045163 -0.561568 1.005155 0.089870 -0.068959
59 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T13:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 8.491595 -0.557745 1.424048 0.089890 -0.066688
60 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T12:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 9.150079 -0.553083 1.949899 0.086221 -0.065420
61 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T11:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 9.390242 -0.549542 1.010224 0.081564 -0.062221
62 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T10:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 9.082260 -0.550943 1.012923 0.077397 -0.055966
63 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T09:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 8.622544 -0.555049 1.014957 0.075107 -0.047349
64 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T08:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.833182 -0.559612 1.015818 0.073993 -0.039062
65 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T07:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 7.119994 -0.564287 1.015359 0.074347 -0.033303
66 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T06:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.824449 -0.568943 1.013925 0.075650 -0.031629
67 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T05:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.660968 -0.571769 1.011828 0.077772 -0.034339
68 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T04:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.740618 -0.569308 2.776619 0.080424 -0.039260
69 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T03:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.696674 -0.566965 4.110346 0.082897 -0.044958
70 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T02:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.547153 -0.568515 6.062504 0.083444 -0.049476
71 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T01:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.359508 -0.571273 6.360367 0.079072 -0.052278
72 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.393537 -0.574618 6.448367 0.068051 -0.051244

73 rows × 13 columns


In [5]:
# Request only data from a specific model run
reftime_end = np.max(df['axes.reftime'])
reftime_start = np.max(df['axes.reftime'])

query_dict = {'apikey': apikey,
              'count': count,
              'lat': lat,
              'lon': lon,
              'reftime_end': reftime_end,
              'reftime_start': reftime_start,
              'time_order': time_order,
              }
query = urlencode(query_dict)

api_url = "http://api.planetos.com/v1/datasets/%s/point?%s" % (id, query)

request = Request(api_url)
response = urlopen(request)
response_json = json.loads(response.read())

data = response_json['entries']

# let's flatten the response and create a Pandas dataframe
df = pd.io.json.json_normalize(data)

# then index by time using the axes.time column
pd.to_datetime(df["axes.time"])
df.set_index('axes.time', inplace=False)

print(df.count())


axes.latitude                            73
axes.longitude                           73
axes.reftime                             73
axes.time                                73
classifiers.reference_time               73
context                                  73
data.ice_coverage                        73
data.ice_thickness                       73
data.mixed_layer_thickness               73
data.ssh                                 73
data.surface_boundary_layer_thickness    73
data.u_barotropic_velocity               73
data.v_barotropic_velocity               73
dtype: int64

All available model run values for a specific time

This example looks for data at a specific future datetime (e.g. 2016-11-14T00:00:00) and returns values from all available model runs (e.g. reftimes). From the result at the time of running, we can see that two forecasts provide values for Nov 14th at 00:00, the 2016-11-11T00:00:00 and 2016-11-12T00:00:00 model runs.


In [ ]:


In [6]:
# Request only data from past model run (e.g. 2016-11-11T00:00:00)

end = np.min(df['axes.time'])
start = datetime.datetime.strftime(datetime.datetime.strptime(df['axes.reftime'][0],'%Y-%m-%dT%H:%M:%S') - datetime.timedelta(days=1),'%Y-%m-%dT%H:%M:%S')

query_dict = {'apikey': apikey,
              'count': count,
              'end': end,
              'lat': lat,
              'lon': lon,
              'reftime_recent': 'false',
              'start': start,
              'time_order': time_order,
              }
query = urlencode(query_dict)

api_url = "http://api.planetos.com/v1/datasets/%s/point?%s" % (id, query)

request = Request(api_url)
response = urlopen(request)
response_json = json.loads(response.read())

data = response_json['entries']

# let's flatten the response and create a Pandas dataframe
df = pd.io.json.json_normalize(data)
print (df)
# then index by time using the axes.time column
pd.to_datetime(df["axes.time"])
df.set_index('axes.time', inplace=True)

print(df.count())
df.head()


   axes.latitude  axes.longitude         axes.reftime            axes.time  \
0      39.390355      -72.616137  2018-06-15T00:00:00  2018-06-15T00:00:00   

  classifiers.reference_time                    context  data.ice_coverage  \
0   2018-06-15T00:00:00.000Z  reference_time_MT_lat_lon                0.0   

   data.ice_thickness  data.mixed_layer_thickness  data.ssh  \
0                 0.0                    6.393537 -0.574618   

   data.surface_boundary_layer_thickness  data.u_barotropic_velocity  \
0                               6.448367                    0.068051   

   data.v_barotropic_velocity  
0                   -0.051244  
axes.latitude                            1
axes.longitude                           1
axes.reftime                             1
classifiers.reference_time               1
context                                  1
data.ice_coverage                        1
data.ice_thickness                       1
data.mixed_layer_thickness               1
data.ssh                                 1
data.surface_boundary_layer_thickness    1
data.u_barotropic_velocity               1
data.v_barotropic_velocity               1
dtype: int64
Out[6]:
axes.latitude axes.longitude axes.reftime classifiers.reference_time context data.ice_coverage data.ice_thickness data.mixed_layer_thickness data.ssh data.surface_boundary_layer_thickness data.u_barotropic_velocity data.v_barotropic_velocity
axes.time
2018-06-15T00:00:00 39.390355 -72.616137 2018-06-15T00:00:00 2018-06-15T00:00:00.000Z reference_time_MT_lat_lon 0.0 0.0 6.393537 -0.574618 6.448367 0.068051 -0.051244

In [ ]: