Harvesting ArcGIS REST endpoints

Config

Sensitive information and related configuration, such as CKAN URLs and credentials, are stored in a separate file. To use this workbook on your own CKAN instance, write or append the following contents into a file secret.py in the same directory as this workbook:

ARCGIS = {
  "SLIPFUTURE" : {
    "url": "http://services.slip.wa.gov.au/arcgis/rest/services",
    "folders": ["QC", ...]
 },
 ...
}

Insert your ArcGIS REST endpoints and service names.


In [1]:
import ckanapi 
from datetime import datetime
import json
import os
import requests
from slugify import slugify

from harvest_helpers import *

from secret import CKAN, ARCGIS

print("The ARCGIS REST service endpoint QC lives at {0}".format(ARCGIS["SLIPFUTURE"]["url"]))
print("The catalogue \"ca\" lives at {0}".format(CKAN["ca"]["url"]))
print("The catalogue \"cb\" lives at {0}".format(CKAN["cb"]["url"]))
print("The catalogue \"ct\" lives at {0}".format(CKAN["ct"]["url"]))

## enable one of:
#ckan = ckanapi.RemoteCKAN(CKAN["ct"]["url"], apikey=CKAN["ct"]["key"])
ckan = ckanapi.RemoteCKAN(CKAN["ca"]["url"], apikey=CKAN["ca"]["key"])
#ckan = ckanapi.RemoteCKAN(CKAN["cb"]["url"], apikey=CKAN["cb"]["key"])

print("Using CKAN {0}".format(ckan.address))


The ARCGIS REST service endpoint QC lives at http://services.slip.wa.gov.au/arcgis/rest/services
The catalogue "ca" lives at http://catalogue.alpha.data.wa.gov.au/
The catalogue "cb" lives at http://catalogue.beta.data.wa.gov.au/
The catalogue "ct" lives at http://test.alpha.data.wa.gov.au/
Using CKAN http://catalogue.alpha.data.wa.gov.au/

Setup

From the config, we'll grab the base URL and folders to be harvested.


In [2]:
baseurl = ARCGIS["SLIPFUTURE"]["url"]
folders = ARCGIS["SLIPFUTURE"]["folders"]
print("The base URL {0} contains folders {1}\n".format(baseurl, str(folders)))

services = get_arc_services(baseurl, folders[0])
print("The services in folder {0} are:\n{1}\n".format(str(folders), str(services)))

service_url = services[0]

mrwa = get_arc_servicedict(services[0])
print("Service {0}\ncontains layers and extensions:\n{1}".format(services[0], mrwa))


The base URL http://services.slip.wa.gov.au/arcgis/rest/services contains folders ['QC']

The services in folder ['QC'] are:
[u'http://services.slip.wa.gov.au/arcgis/rest/services/QC/MRWA_Public_Services/MapServer']

Service http://services.slip.wa.gov.au/arcgis/rest/services/QC/MRWA_Public_Services/MapServer
contains layers and extensions:
{'layer_ids': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26'], 'supportedExtensions': u'WFSServer, WMSServer'}

Harvest

Hammertime? Hammertime.


In [3]:
harvest_arcgis_service(services[0],
                       ckan,
                       owner_org_id = ckan.action.organization_show(id="mrwa")["id"],
                       author = "Main Roads Western Australia",
                       author_email = "irissupport@mainroads.wa.gov.au",
                       debug=False)



Parsing layer 0
Writing dataset Signs - Guide...
[upsert_dataset] Reading WMS layer signs-guide
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset Signs - Guide to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 1
Writing dataset RIVN Road Stopping Place...
[upsert_dataset] Reading WMS layer rivn-road-stopping-place
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Road Stopping Place to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 2
Writing dataset NTWK IRIS Road Network...
[upsert_dataset] Reading WMS layer ntwk-iris-road-network
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset NTWK IRIS Road Network to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 3
Writing dataset CLSS Road Hierarchy...
[upsert_dataset] Reading WMS layer clss-road-hierarchy
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset CLSS Road Hierarchy to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 4
Writing dataset RIVN Rail Crossing...
[upsert_dataset] Reading WMS layer rivn-rail-crossing
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Rail Crossing to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 5
Writing dataset RIVN Pedestrian Crossing...
[upsert_dataset] Reading WMS layer rivn-pedestrian-crossing
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Pedestrian Crossing to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 6
Writing dataset RIVN Line Marking...
[upsert_dataset] Reading WMS layer rivn-line-marking
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Line Marking to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 7
Writing dataset RIVN Legal Speed Zones...
[upsert_dataset] Reading WMS layer rivn-legal-speed-zones
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Legal Speed Zones to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 8
Writing dataset RIVN Legal Speed Limit...
[upsert_dataset] Reading WMS layer rivn-legal-speed-limit
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Legal Speed Limit to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 9
Writing dataset Signs - Hazard Markers...
[upsert_dataset] Reading WMS layer signs-hazard-markers
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset Signs - Hazard Markers to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 10
Writing dataset RIVN Signs Other...
[upsert_dataset] Reading WMS layer rivn-signs-other
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Signs Other to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 11
Writing dataset RIVN Signs Regulatory...
[upsert_dataset] Reading WMS layer rivn-signs-regulatory
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Signs Regulatory to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 12
Writing dataset RIVN Signs Warning...
[upsert_dataset] Reading WMS layer rivn-signs-warning
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Signs Warning to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 13
Writing dataset NTWK State Road Network Future...
[upsert_dataset] Reading WMS layer ntwk-state-road-network-future
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset NTWK State Road Network Future to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 14
Writing dataset STRT Structures...
[upsert_dataset] Reading WMS layer strt-structures
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset STRT Structures to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 15
Writing dataset TRFF Traffic Digest...
[upsert_dataset] Reading WMS layer trff-traffic-digest
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset TRFF Traffic Digest to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 16
Writing dataset Traffic Signals...
[upsert_dataset] Reading WMS layer traffic-signals
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset Traffic Signals to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 17
Writing dataset RIVN Wall Type or Fence...
[upsert_dataset] Reading WMS layer rivn-wall-type-or-fence
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Wall Type or Fence to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 18
Writing dataset NTWK Intersections...
[upsert_dataset] Reading WMS layer ntwk-intersections
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset NTWK Intersections to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 19
Writing dataset RIVN Kerb...
[upsert_dataset] Reading WMS layer rivn-kerb
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Kerb to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 20
Writing dataset CLSS Functional Class...
[upsert_dataset] Reading WMS layer clss-functional-class
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset CLSS Functional Class to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 21
Writing dataset RIVN Focal Point Marker...
[upsert_dataset] Reading WMS layer rivn-focal-point-marker
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Focal Point Marker to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 22
Writing dataset RIVN Floodway...
[upsert_dataset] Reading WMS layer rivn-floodway
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Floodway to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 23
Writing dataset RIVN Culvert...
[upsert_dataset] Reading WMS layer rivn-culvert
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Culvert to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 24
Writing dataset TRFF Bike Count Sites...
[upsert_dataset] Reading WMS layer trff-bike-count-sites
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset TRFF Bike Count Sites to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 25
Writing dataset RIVN Barrier...
[upsert_dataset] Reading WMS layer rivn-barrier
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset RIVN Barrier to CKAN http://catalogue.alpha.data.wa.gov.au/


Parsing layer 26
Writing dataset Control of Access on State Roads ...
[upsert_dataset] Reading WMS layer control-of-access-on-state-roads
[upsert_dataset]  Layer exists.
  [upsert_dataset]  Existing dataset metadata were updated.
  [upsert_dataset]  Existing resources were replaced with new resources.
Upserted dataset Control of Access on State Roads to CKAN http://catalogue.alpha.data.wa.gov.au/

In [5]:
a = ["SLIP Classic", "Harvested"]
a.append("test")
a


Out[5]:
['SLIP Classic', 'Harvested', 'test']

In [ ]: