Datahub Metadata Query Example

This notebook outlines how to capture a list of available datasets using the Datahub API. The output here is quite simple, but the avaialble metadata keys are displayed and output can be tailored to include additional attributes as needed.

API documentation is available at http://docs.planetos.com. If you have questions or comments, join our Slack community to chat with our development team.

For general information on usage of IPython/Jupyter and Matplotlib, please refer to their corresponding documentation. https://ipython.org/ and http://matplotlib.org/


In [1]:
from urllib.request import urlopen, Request
from urllib.parse import urlencode
import simplejson as json

Important! You'll need to replace apikey below with your actual Planet OS API key, which you'll find on the Planet OS account settings page.


In [26]:
apikey = open('APIKEY').readlines()[0].strip() #'<YOUR API KEY HERE>'
url = "http://api.planetos.com/v1/datasets?apikey=%s" % apikey
print(url)
request = Request(url)
response = urlopen(request)
dataset_ids = json.loads(response.read())


http://api.planetos.com/v1/datasets?apikey=8428878e4b944abeb84790e832c633fc

In [27]:
# inspect the available keys and metadata structure from a single dataset
api_endpoint = "http://api.planetos.com/v1/datasets/"
query_dict = {'apikey': apikey}
query = urlencode(query_dict)

datasets = []

for id in dataset_ids:
    api_query = "%s%s?%s" % (api_endpoint, id, query)
    request = Request(api_query)
    response = urlopen(request)
    response_json = json.loads(response.read())

    datasets.append(response_json)

datasets[0].keys()


Out[27]:
dict_keys(['TemporalExtentEnd', 'ExampleApiParameters', 'UpdateFrequency', 'Title', 'OnlineResource', 'Licence', 'Institution', 'VerticalExtent', 'TemporalResolution', 'Categories', 'Description', 'ProductType', 'Refreshed', 'SpatialExtent', 'PreviewOptions', 'SpatialReferenceSystem', 'SpatialExtentText', 'Abstract', 'TemporalExtentStart', 'Source', 'LicenceType', 'Variables', 'SpatialResolution', 'FeatureType', 'Annotations', 'Publisher', 'DistributionFormat', 'DataVendorKey', 'Key'])

In [ ]:


In [28]:
# Alphabetically sort by dataset title
alpha_datasets = sorted(datasets, key=lambda k: k['Title'])

print("%s Datasets Available on Datahub\nhttp://data.planetos.com/\n" % len(datasets))

for ds in alpha_datasets:
    datahub_url = "http://data.planetos.com/datasets/%s" % ds['Key']
    # output Title, publisher, and Datahub url
    print("(%s) %s\n%s\n" % (ds['Publisher'], ds['Title'], datahub_url))
    # output Title, publisher, Datahub url and full abstract
#     print("%s (%s)\n%s\n--\n%s\n" % (ds['Publisher'], ds['Title'], datahub_url, ds['Abstract']))


82 Datasets Available on Datahub
http://data.planetos.com/

(Bureau of Meteorology, Australia)  Bureau of Meteorology (BOM) Daily Rain, Temperature and Vapour Pressure for Australia
http://data.planetos.com/datasets/bom_clim_australia

(Bureau of Meteorology, Australia)  Bureau of Meteorology (BOM) Daily solar exposure for Australia
http://data.planetos.com/datasets/bom_solar_australia

(Bureau of Meteorology, Australia)  Bureau of Meteorology (BOM) Monthly NDVI Average for Australia
http://data.planetos.com/datasets/bom_ndvi_australia

(Copernicus)  Copernicus Atmosphere Monitoring Service (CAMS) global near-real-time production system
http://data.planetos.com/datasets/cams_nrt_forecasts_global

(NOAA) 3-day cumulative USF AFAI Fields
http://data.planetos.com/datasets/noaa_aoml_afai

(Copernicus Marine Environment Monitoring System (CMEMS)) Baltic Sea - in situ near real time observations
http://data.planetos.com/datasets/cmems_baltic_observations_daily

(Copernicus) Baltic Sea Biogeochemistry Analysis and Forecast
http://data.planetos.com/datasets/copernicus_biogeo_baltic_hourly

(MyOcean) Baltic Sea: Sea Surface Temperature Analysis
http://data.planetos.com/datasets/myocean_sst_baltic_daily

(NOAA) Blended Sea Winds
http://data.planetos.com/datasets/noaa_blended_sea_winds_6hr_global_0.25d

(NOAA) Blended Sea Winds Climatological Monthlies
http://data.planetos.com/datasets/noaa_blended_sea_winds_clim_global

(Remote Sensing Systems) CCMP Wind Vector Analysis Product V2.0
http://data.planetos.com/datasets/rss_ccmp_winds_v2

(NOAA) CFS Operational 9-Month forecast
http://data.planetos.com/datasets/ncep_cfsv2

(NOAA) CPC Unified Gauge-Based Analysis of Daily Precipitation over CONUS
http://data.planetos.com/datasets/noaa_urd_conus

(Climate Hazards Group (CHG)) Climate Hazards Group InfraRed Precipitation with Station data (CHIRPS) Version 2.0
http://data.planetos.com/datasets/chg_chirps_global_05

(DWD) DWD WAve Model Europe
http://data.planetos.com/datasets/dwd_wam_europe

(DWD) DWD WAve Model Global
http://data.planetos.com/datasets/dwd_wam_global

(Ifremer) Daily ASCAT Surface Wind Fields 
http://data.planetos.com/datasets/ifremer_ascat_daily

(Copernicus) ECMWF Fire Danger Forecast Reanalysis
http://data.planetos.com/datasets/ecmwf_fire_danger_forecast_reanalysis

(ECMWF) ERA5
http://data.planetos.com/datasets/ecmwf_era5

(NOAA) ETOPO Global Relief Model
http://data.planetos.com/datasets/noaa_etopo_global_1arcmin

(MyOcean) European Ocean Sea Surface Temperature Multi-Sensor L3 Observations
http://data.planetos.com/datasets/myocean_sst_europe_daily

(Finnish Meteorological Institute (FMI)) FMI Hirlam Weather Model
http://data.planetos.com/datasets/fmi_hirlam_surface

(Finnish Meteorological Institute (FMI)) Finnish Meteorological Institute - System for Integrated modeLling of Atmospheric coMposition (SILAM)
http://data.planetos.com/datasets/fmi_silam_global05

(NOAA) GFS Analysis
http://data.planetos.com/datasets/noaa_gfs_pgrb2_global_analysis_0.25degree

(NOAA) GFS forecast
http://data.planetos.com/datasets/noaa_gfs_pgrb2_global_forecast_recompute_0.25degree

(NOAA) GFS global weather forecast by NCEP. Near surface parameters.
http://data.planetos.com/datasets/noaa_gfs_global_sflux_0.12d

(NASA) GHRSST Level 4 G1SST Global Foundation Sea Surface Temperature Analysis
http://data.planetos.com/datasets/nasa_ghrsst_global_daily

(Deutsche Wetterdienst (DWD)) GPCC first guess daily, version 1.0, precipitation per grid in mm/day
http://data.planetos.com/datasets/gpcc_first_guess_daily

(NASA) GRACE Tellus Monthly Mass Grids - Land
http://data.planetos.com/datasets/nasa_grctellus_land

(NASA) GRACE Tellus Monthly Mass Grids - Ocean
http://data.planetos.com/datasets/nasa_grctellus_ocean

(NOAA) Global Ensemble Forecast System (GEFS)
http://data.planetos.com/datasets/noaa-ncep_gefs

(NOAA) Global Ensemble Forecast System (GEFS) average fields
http://data.planetos.com/datasets/noaa-ncep_gefs_averages

(NASA) Global Land Data Assimilation System (GLDAS) Land Water Content (Monthly)
http://data.planetos.com/datasets/nasa_gldas_lwc_monthly

(Copernicus Marine Environment Monitoring System (CMEMS)) Global Observed Ocean Physics Temperature Salinity Heights and Currents
http://data.planetos.com/datasets/cmems_global_analysis_phys_001_020

(Copernicus) Global Ocean Biogeochemical Analysis and Forecast
http://data.planetos.com/datasets/copernicus_goba_global_weekly

(Copernicus Marine Environment Monitoring System (CMEMS)) Global Ocean Wind L4 Near Real Time 6 Hourly Observations V5
http://data.planetos.com/datasets/cmems_gwind

(NOAA) Global Real-Time Ocean Forecast System (RTOFS) - 2D Surface 1 Hourly Diagnostic Forecast
http://data.planetos.com/datasets/noaa_rtofs_surface_1h_diag

(Bureau of Meteorology, Australia) Global domain of the Australian Community Climate and Earth-System Simulator (ACCESS-G), numerical weather prediction (NWP) data.
http://data.planetos.com/datasets/bom_access-g_global_40km

(HYCOM consortium) HYCOM (Hybrid Coordinate Ocean Model) global ocean forecast
http://data.planetos.com/datasets/hycom_glbu0.08_91.2_global_0.08d

(NOAA) High-Resolution Rapid Refresh (HRRR) - CONUS 3km, 2D surface Levels
http://data.planetos.com/datasets/noaa_hrrr_surface_hourly

(NASA) Integrated Multi-satellitE Retrievals for GPM (IMERG) - Day 1 Early Run Half-Hourly Product
http://data.planetos.com/datasets/nasa_3imerghhe

(NASA) Integrated Multi-satellitE Retrievals for GPM (IMERG) - Day 1 Late Run Half-Hourly Product
http://data.planetos.com/datasets/nasa_3imerghhl

(NOAA) International Comprehensive Ocean-Atmosphere Data Set, Monthly Summary Statistics, 1°
http://data.planetos.com/datasets/noaa_icoads_enhanced_1d_day

(Met.no) MET Norway HARMONIE forecast
http://data.planetos.com/datasets/metno_harmonie_metcoop

(Met.no) MET Norway HARMONIE wind forecast
http://data.planetos.com/datasets/metno_harmonie_wind_det

(U.S. Geological Survey (USGS)) MODIS Monthly Vegetation Indices Level 3 Data
http://data.planetos.com/datasets/usgs_vegetation_global

(Copernicus Marine Environment Monitoring System (CMEMS)) Mediterranean Sea Waves Hindcast and Forecast
http://data.planetos.com/datasets/cmems_medsea_wave_analysis_forecast_0042

(Meteo-France) MeteoFrance - AROME Forecast
http://data.planetos.com/datasets/meteofrance_arome_001_surface

(Meteo-France) MeteoFrance - MFWAM wave model
http://data.planetos.com/datasets/meteofrance_mfwam_0025_france

(NOAA) Meteorological observations from Regional Basic Synoptic Network (RBSN) Version 1.0
http://data.planetos.com/datasets/noaa_rbsn_timeseries

(National Snow & Ice Data Center (NSIDC)) Modis/Terra Snow Cover Daily L3 Global 
http://data.planetos.com/datasets/nsidc_modis_snow_cover_global

(NCEP) NCEP Climate Forecast System Reanalysis (CFSR)
http://data.planetos.com/datasets/ncep_cfsr_global_03

(NOAA) NCEP MMAB Global Visibility and Ice Accretion Guidance based on GFS Model Output
http://data.planetos.com/datasets/ncep_ice_fog_global

(NOAA) NDBC Continuous Winds data
http://data.planetos.com/datasets/noaa_ndbc_cwind_stations

(NOAA) NDBC MMS Acoustic Doppler Current Profiler data
http://data.planetos.com/datasets/noaa_ndbc_adcp_station

(NOAA) NDBC Spectral Wave Density data with Spectral Wave Direction data
http://data.planetos.com/datasets/noaa_ndbc_swden_stations

(NOAA) NDBC Standard Meteorological data
http://data.planetos.com/datasets/noaa_ndbc_stdmet_stations

(NOAA) NOAA 1/4° daily Optimum Interpolation Sea Surface Temperature (daily OISST) 
http://data.planetos.com/datasets/noaa_oisst_daily_1_4

(NOAA) NOAA Wave Watch III (NWW3) Ocean Wave Model
http://data.planetos.com/datasets/noaa_ww3_global_1.25x1d

(NOAA) NOAA WaveWatch III Regional Alaska (4 arc-minute) Wave Model
http://data.planetos.com/datasets/noaa_ww3_ak

(NOAA) NOAA WaveWatch III Regional US West Coast (4 arc-minute) Wave Model
http://data.planetos.com/datasets/noaa_ww3_wc

(NOAA) National Air Quality Forecast System (AQFS): 1-Hour Average Ozone CONUS
http://data.planetos.com/datasets/noaa_aqfs_avg_1h_o3_conus

(NOAA) National Air Quality Forecast System (AQFS): Bias Corrected PM2.5 CONUS
http://data.planetos.com/datasets/noaa_aqfs_pm25_bc_conus

(SOCIB) Near real-time surface current data in the Ibiza Channel
http://data.planetos.com/datasets/socib_hfradar_ibiza_hourly

(NOAA) North American Mesoscale Forecast System (NAM): Alaska Nest - 6 km
http://data.planetos.com/datasets/noaa_nam_alaska

(NOAA) North American Mesoscale Forecast System (NAM): CONUS Full Complement of Pressure Level Fields - 12 km
http://data.planetos.com/datasets/noaa_nam_awips_phys

(NOAA) North American Mesoscale Forecast System (NAM): CONUS Full Complement of Surface-Based Fields - 12 km
http://data.planetos.com/datasets/noaa_nam_awips_12

(NOAA) North American Mesoscale Forecast System (NAM): CONUS Nest - 5 km
http://data.planetos.com/datasets/noaa_nam_conusnest

(NOAA) North American Mesoscale Forecast System (NAM): Central America/Caribbean - 12 km
http://data.planetos.com/datasets/noaa_nam_ca

(NOAA) North American Mesoscale Forecast System (NAM): Hawaii Nest - 3 km
http://data.planetos.com/datasets/noaa_nam_hawaii

(NOAA) North American Mesoscale Forecast System (NAM): North Pacific - 12 km
http://data.planetos.com/datasets/noaa_nam_north_pacific

(NOAA) North American Mesoscale Forecast System (NAM): Puerto Rico Nest - 3 km
http://data.planetos.com/datasets/noaa_nam_prico

(NASA) OSCAR (Ocean Surface Current Analysis Real-Time) Ocean Surface Currents 1/3 Degree
http://data.planetos.com/datasets/nasa_oscar_global_5day

(Pacific Islands Ocean Observing System (PacIOOS)) Simulating WAves Nearshore (SWAN) Regional Wave Model: Oahu
http://data.planetos.com/datasets/pacioos_swan_oahu

(SOCAT) Socat v4 - Surface Ocean CO2 Variability and Vulnerability
http://data.planetos.com/datasets/socat_v4_yearly

(Climate Prediction Center) The Africa Rainfall Climatology version 2 (ARC2)
http://data.planetos.com/datasets/noaa_arc2_africa_01

(Canadian Meteorological Centre (CMC)) The Global Deterministic Wave Prediction System (GDWPS) wave model
http://data.planetos.com/datasets/cmc_gdwps_wave_model_global

(NASA) The Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2)
http://data.planetos.com/datasets/nasa_merra2_global

(SOCIB) The Western Mediterranean Operational forecasting system (WMOP)
http://data.planetos.com/datasets/socib_forecast_western_mediterranean_daily

(NOAA) WPC Quantitative Precipitation Forecast (QPF) 7 days
http://data.planetos.com/datasets/noaa_wpc_qpf_1_7_days_6hr

(NOAA) Wavewatch III global model
http://data.planetos.com/datasets/noaa_ww3_global_history

(NOAA) Weekly mean carbon dioxide measured at Mauna Loa Observatory, Hawaii
http://data.planetos.com/datasets/noaa_co2_obs_weekly


In [ ]: