Exploring CSW access in Python using OWSLib with Data.gov



In [1]:

    
from pylab import *
from owslib.csw import CatalogueServiceWeb



In [2]:

    
from IPython.core.display import HTML
HTML('<iframe src=http://geo.gov.ckan.org/dataset width=1024 height=600></iframe>')









    Out[2]:



In [3]:

    
# connect to CSW, explore it's properties
#endpoint = 'http://www.ngdc.noaa.gov/geoportal/csw' # NGDC Geoportal
#endpoint = 'http://www.nodc.noaa.gov/geoportal/csw'   # NODC Geoportal: granule level
#endpoint = 'http://data.nodc.noaa.gov/geoportal/csw'  # NODC Geoportal: collection level
    
#endpoint = 'http://geodiscover.cgdi.ca/wes/serviceManagerCSW/csw'  # NRCAN CUSTOM
#endpoint = 'http://geoport.whoi.edu/gi-cat/services/cswiso' # USGS Woods Hole GI_CAT
#endpoint = 'http://cida.usgs.gov/gdp/geonetwork/srv/en/csw' # USGS CIDA Geonetwork

#endpoint = 'http://geoport.whoi.edu/geoportal/csw'
endpoint = 'http://geo.gov.ckan.org/csw'

csw = CatalogueServiceWeb(endpoint,timeout=30)
csw.version









    Out[3]:





'2.0.2'



In [4]:

    
[op.name for op in csw.operations]









    Out[4]:





['GetCapabilities',
 'GetRepositoryItem',
 'DescribeRecord',
 'GetDomain',
 'GetRecordById',
 'GetRecords']



In [5]:

    
# Perform the CSW query.  To get the Data URLS (including DAP), we need to specify
# esn='full' to get the full Dublin Core response ('summary' is the default)

bbox=[-71.5, 39.5, -63.0, 46]
#bbox=[-180, 0, 180.0, 90]
keywords=['temperature','netcdf']
maxrecords = 10

csw.getrecords(keywords=keywords,bbox=bbox,maxrecords=maxrecords,esn='full')
csw.results









    Out[5]:





{'matches': 634, 'nextrecord': 11, 'returned': 10}



In [6]:

    
for rec,item in csw.records.iteritems():
    print item.title









    



Mass Spectrometer Incoherent Scatter (MSIS) Model
RSS SSM/I Ocean Product Grids Daily from DMSP F15 netCDF
Surface Turbulent Fluxes, 1x1 deg Daily Grid, Satellite F11
MODIS/Terra Sea Ice Extent Daily L3 Global 1km EASE-Grid Night V003
GASP
UARS Correlative UKMO Daily Gridded Stratospheric Assimilated Data
AIRS/Aqua Near Real Time (NRT) Level 1B Visible/Near Infrared (VIS/NIR) geolocated and calibrated radiances
Global Population Density
AVHRR
MODIS/Terra Thermal Anomalies/Fire 8-Day L3 Global 1km SIN Grid V005



In [7]:

    
for rec,item in csw.records.iteritems():
    print item.references









    



[{'url': 'http://nssdc.gsfc.nasa.gov/', 'scheme': 'None'}]
[{'url': 'http://ghrc.nsstc.nasa.gov/', 'scheme': 'None'}, {'url': 'http://reverb.echo.nasa.gov/reverb/#utf8=%E2%9C%93&spatial_map=satellite&spatial_type=rectangle&keywords=rssmif15d', 'scheme': 'None'}]
[{'url': 'http://disc.gsfc.nasa.gov/', 'scheme': 'None'}, {'url': 'http://mirador.gsfc.nasa.gov/', 'scheme': 'None'}, {'url': 'ftp://meso-a.gsfc.nasa.gov/pub/shieftp/fluxdata/', 'scheme': 'None'}, {'url': 'ftp://measures.gsfc.nasa.gov/data/s4pa/GSSTF/GSSTF_F11.2b/doc/README.GSSTF2b.pdf', 'scheme': 'None'}]
[{'url': 'http://nsidc.org/daac/index.html', 'scheme': 'None'}, {'url': 'http://nsidc.org', 'scheme': 'None'}, {'url': 'http://nsidc.org/modis/', 'scheme': 'None'}]
[{'url': 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=GASP&amp;view_id=map', 'scheme': 'None'}, {'url': 'http://webapps.datafed.net/wms_query.aspx?dataset_abbr=GASP&amp;view_id=map', 'scheme': 'None'}]
[{'url': 'http://disc.gsfc.nasa.gov/', 'scheme': 'None'}, {'url': 'http://mirador.gsfc.nasa.gov/cgi-bin/mirador/presentNavigation.pl?tree=project&project=UARS&CGISESSID=7e79a4cca5f3609012cd9a727d63ad1d', 'scheme': 'None'}, {'url': 'ftp://acdisc.gsfc.nasa.gov/data/s4pa/UARS_Correlative_Level4/UARZCUKM/doc', 'scheme': 'None'}]
[{'url': 'http://disc.gsfc.nasa.gov/', 'scheme': 'None'}, {'url': 'http://disc.sci.gsfc.nasa.gov/acdisc/', 'scheme': 'None'}, {'url': 'ftp://discnrt1.gesdisc.eosdis.nasa.gov/ftp/data/Aqua_NRT/AIRVBRAD_NRT.005/', 'scheme': 'None'}, {'url': 'http://discnrt1.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_NRT/AIRVBRAD_NRT.005/contents.html', 'scheme': 'None'}, {'url': 'http://discnrt1.gesdisc.eosdis.nasa.gov/datacasting/AIRVBRAD_NRT.005.datacast-feed.xml', 'scheme': 'None'}, {'url': 'http://www-airs.jpl.nasa.gov/', 'scheme': 'None'}, {'url': 'http://disc.gsfc.nasa.gov/AIRS/index.shtml', 'scheme': 'None'}, {'url': 'http://disc.sci.gsfc.nasa.gov/AIRS/data-holdings/by-data-product/AIRVBRAD_NRT.shtml', 'scheme': 'None'}, {'url': 'http://disc.sci.gsfc.nasa.gov/nrt/data-holdings/airs-nrt-products', 'scheme': 'None'}, {'url': 'ftp://airsl1.gesdisc.eosdis.nasa.gov/ftp/data/s4pa/Aqua_AIRS_Level1/AIRVBRAD.005/doc/README.AIRVBRAD.pdf', 'scheme': 'None'}]
[{'url': 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=CIESIN_GPW&amp;view_id=map', 'scheme': 'None'}, {'url': 'http://webapps.datafed.net/wms_query.aspx?dataset_abbr=CIESIN_GPW&amp;view_id=map', 'scheme': 'None'}]
[{'url': 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=AVHRR&amp;view_id=map', 'scheme': 'None'}, {'url': 'http://webapps.datafed.net/wms_query.aspx?dataset_abbr=AVHRR&amp;view_id=map', 'scheme': 'None'}]
[{'url': 'https://lpdaac.usgs.gov/', 'scheme': 'None'}, {'url': 'http://reverb.echo.nasa.gov/reverb/#utf8=%E2%9C%93&spatial_map=satellite&spatial_type=rectangle&keywords=MOD14A25', 'scheme': 'None'}, {'url': 'http://edcdaac.usgs.gov/', 'scheme': 'None'}, {'url': 'http://modis-land.gsfc.nasa.gov/', 'scheme': 'None'}]



In [8]:

    
# function to get specific ServiceType URL from records
def service_urls(records,service_string='urn:x-esri:specification:ServiceType:odp:url'):
    urls=[]
    for key,rec in records.iteritems():
        #create a generator object, and iterate through it until the match is found
        #if not found, gets the default value (here "none")
        url = next((d['url'] for d in rec.references if d['scheme'] == service_string), None)
        if url is not None:
            urls.append(url)
    return urls



In [9]:

    
urls = service_urls(csw.records,service_string='None')
print(urls)
len(urls)









    



['http://nssdc.gsfc.nasa.gov/', 'http://ghrc.nsstc.nasa.gov/', 'http://disc.gsfc.nasa.gov/', 'http://nsidc.org/daac/index.html', 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=GASP&amp;view_id=map', 'http://disc.gsfc.nasa.gov/', 'http://disc.gsfc.nasa.gov/', 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=CIESIN_GPW&amp;view_id=map', 'http://webapps.datafed.net/wcs_query.aspx?dataset_abbr=AVHRR&amp;view_id=map', 'https://lpdaac.usgs.gov/']






    Out[9]:





10



In [10]:

    
# examine the 1st record
a=csw.records.keys()
foo=csw.records[a[0]]



In [11]:

    
foo.abstract









    Out[11]:





"The Mass-Spectrometer-Incoherent-Scatter (MSIS) model describes the neutral temperature and densities in the Upper Atmosphere (above about 100 km). MSIS-86 constitutes the upper part of the COSPAR International Reference Atmosphere (CIRA) 1986. The MSIS model is based on the extensive data compilation and analysis work of A.E. Hedin (GSFC, code 614) and his colleagues. Data sources include measurements from several rockets, satellites (OGO-6, San Marco 3, AEROS-A, AE-C, AE-D, AE-E, ESRO 4, and DE-2), and incoherent scatter radars (Millstone Hill, St. Santin, Arecibo, Jicamarca, and Malvern). The model expects as input: year, day of year, universal time, altitude, geodetic latitude and longitude, local apparent solar time, solar F10.7 flux (for previous day and 3-month average), and magnetic Ap index (daily or Ap history for last 59 hours). For these conditions the following output parameters are calculated: number density of He, O, N2, O2, Ar, H and N, total mass density, neutral temperature and exospheric temperature. For diagnostic purposes the source code is equipped with 23 flags to turn on/off particular variations. The model software is available from NSSDC on magnetic tape, diskette for use on PCs, and online over SPAN. The MSIS model can also be run online interactively on NSSDC's NODIS account."



In [12]:

    
foo.references









    Out[12]:





[{'scheme': 'None', 'url': 'http://nssdc.gsfc.nasa.gov/'}]



In [13]:

    
foo.xml









    Out[13]:





'<csw:Record xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" xmlns:gml="http://www.opengis.net/gml" xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ows="http://www.opengis.net/ows" xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:dct="http://purl.org/dc/terms/" xmlns:ogc="http://www.opengis.net/ogc"><dc:identifier>ac33f735-a601-4c11-a24e-d9e9d138457b</dc:identifier><dc:title>Mass Spectrometer Incoherent Scatter (MSIS) Model</dc:title><dc:type>dataset</dc:type><dc:subject>EARTH SCIENCE &gt; ATMOSPHERE &gt; ATMOSPHERIC TEMPERATURE &gt; AIR TEMPERATURE</dc:subject><dc:subject>EARTH SCIENCE &gt; SUN-EARTH INTERACTIONS &gt; SOLAR ENERGETIC PARTICLE PROPERTIES &gt; PARTICLE COMPOSITION</dc:subject><dc:subject>EARTH SCIENCE &gt; SUN-EARTH INTERACTIONS &gt; SOLAR ENERGETIC PARTICLE PROPERTIES &gt; PARTICLE DENSITY</dc:subject><dc:subject>EARTH SCIENCE &gt; SUN-EARTH INTERACTIONS &gt; SOLAR ENERGETIC PARTICLE PROPERTIES &gt; PARTICLE TEMPERATURE</dc:subject><dc:subject>EARTH SCIENCE &gt; SOLID EARTH &gt; GEOMAGNETISM &gt; GEOMAGNETIC INDICES</dc:subject><dc:subject>EARTH SCIENCE &gt; SUN-EARTH INTERACTIONS &gt; IONOSPHERE/MAGNETOSPHERE DYNAMICS &gt; GEOMAGNETIC INDICES</dc:subject><dc:subject>Idn_node Gsfc/nssdc</dc:subject><dc:subject>Msis</dc:subject><dc:subject>Thermosphere</dc:subject><dc:subject>CLIMATOLOGY/METEOROLOGY/ATMOSPHERE</dc:subject><dc:subject>GEOSCIENTIFIC INFORMATION</dc:subject><dct:references scheme="None">http://nssdc.gsfc.nasa.gov/</dct:references><dct:abstract>The Mass-Spectrometer-Incoherent-Scatter (MSIS) model describes the neutral temperature and densities in the Upper Atmosphere (above about 100 km). MSIS-86 constitutes the upper part of the COSPAR International Reference Atmosphere (CIRA) 1986. The MSIS model is based on the extensive data compilation and analysis work of A.E. Hedin (GSFC, code 614) and his colleagues. Data sources include measurements from several rockets, satellites (OGO-6, San Marco 3, AEROS-A, AE-C, AE-D, AE-E, ESRO 4, and DE-2), and incoherent scatter radars (Millstone Hill, St. Santin, Arecibo, Jicamarca, and Malvern). The model expects as input: year, day of year, universal time, altitude, geodetic latitude and longitude, local apparent solar time, solar F10.7 flux (for previous day and 3-month average), and magnetic Ap index (daily or Ap history for last 59 hours). For these conditions the following output parameters are calculated: number density of He, O, N2, O2, Ar, H and N, total mass density, neutral temperature and exospheric temperature. For diagnostic purposes the source code is equipped with 23 flags to turn on/off particular variations. The model software is available from NSSDC on magnetic tape, diskette for use on PCs, and online over SPAN. The MSIS model can also be run online interactively on NSSDC\'s NODIS account.</dct:abstract><dc:date>9999-01-01</dc:date><dc:language>eng; USA</dc:language><dc:rights>otherRestrictions</dc:rights><ows:BoundingBox crs="urn:x-ogc:def:crs:EPSG:6.11:4326" dimensions="2"><ows:LowerCorner>-90.0 -180.0</ows:LowerCorner><ows:UpperCorner>90.0 180.0</ows:UpperCorner></ows:BoundingBox></csw:Record>'



In [13]: