Go down for licence and other metadata about this presentation
\newpage
\newpage
In [2]:
from IPython.display import YouTubeVideo
YouTubeVideo('F4rFuIb1Ie4')
Out[2]:
In [34]:
## PDF output using pandoc
import os
### Export this notebook as markdown
commandLineSyntax = 'ipython nbconvert --to markdown 20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb'
print (commandLineSyntax)
os.system(commandLineSyntax)
### Export this notebook and the document header as PDF using Pandoc
commandLineSyntax = 'pandoc -f markdown -t latex -N -V geometry:margin=1in DocumentHeader.md 20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.md --filter pandoc-citeproc --latex-engine=xelatex --toc -o interim.pdf '
os.system(commandLineSyntax)
### Remove cruft from the pdf
commandLineSyntax = 'pdftk interim.pdf cat 1-5 18-end output 20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.pdf'
os.system(commandLineSyntax)
### Remove the interim pdf
commandLineSyntax = 'rm interim.pdf'
os.system(commandLineSyntax)
Out[34]:
In [3]:
%install_ext https://raw.githubusercontent.com/rasbt/python_reference/master/ipython_magic/watermark.py
%load_ext watermark
In [4]:
%watermark -a "Anthony Beck" -d -v -m -g
In [5]:
#List of installed conda packages
!conda list
In [8]:
#List of installed pip packages
!pip list
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
In [ ]:
# Notes don't show in a python3 environment
!ipython nbconvert 20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb --to slides --post serve
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
In [13]:
#Future proof python 2
from __future__ import print_function #For python3 print syntax
from __future__ import division
# def
import IPython.core.display
# A function to collect user input - ipynb_input(varname='username', prompt='What is your username')
def ipynb_input(varname, prompt=''):
"""Prompt user for input and assign string val to given variable name."""
js_code = ("""
var value = prompt("{prompt}","");
var py_code = "{varname} = '" + value + "'";
IPython.notebook.kernel.execute(py_code);
""").format(prompt=prompt, varname=varname)
return IPython.core.display.Javascript(js_code)
# inline
%pylab inline
\newpage
\newpage
This presentation contributes to the following learning outcomes for this course.
and the word was cartography
\newpage
\newpage
\newpage
Restrictive data
\newpage
Disconnected data with different:
\newpage
\newpage
In [2]:
from IPython.display import YouTubeVideo
YouTubeVideo('xew6qI-6wNk')
Out[2]:
\newpage
\newpage
Making data interoperable and open
\newpage
\newpage
\newpage
the difference in data format. The same logical model can be represented in a range of different physical models (for example ESRI shape file or Geography Mark-up Language (GML)).
This mismatch between underlying data models implies that the same information could be represented differently in different organisations.
The most profound difference is in the storage paradigm:
@beck_uk_2008, @bishr_overcoming_1998
\newpage
Semantic heterogeneity refers to differences in naming conventions and conceptual groupings.
This can be subdivided into naming and cognitive heterogeneities.
@beck_uk_2008, @bishr_overcoming_1998
\newpage
refers to the differences in data model between organisations modelling the same concepts.
This reflects each organisation’s abstracted view of their business and physical assets. Hence, different hierarchical and classification concepts are adopted by each organisation to refer to identical or similar real world objects.
@beck_uk_2008, @bishr_overcoming_1998
\newpage
\newpage
OGC’s Open Standards are:
OGC’s Open Standards are submitted to other industry and National Standards Development Organisations in the vertical area and to global organisations like ISO for standard branding.
\newpage
\newpage
\newpage
In [23]:
from IPython.display import IFrame
IFrame('http://www.opengeospatial.org/standards', width=1000, height=700)
Out[23]:
\newpage
\newpage
From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage
\newpage
\newpage
\newpage
\newpage
\newpage
Open access to knowledge gained significant momementum with the increased uptake of the World Wide Web. This is particularly seen in initiatives like Wikipedia (established in 2001) and Open Knowledge (was the Open Knowledge Foundation: established in 2004). Within the Geo community Open Street Map (also established in 2004) and the Open Source Geospatial Foundation (OSGeo - established in 2006) are key initiatives that promote accessible data and software resources respectively.
Critical to this is that these were grass roots (community) movements that have proven to be highly disruptive to incumbent data providers, practices and policies.
\newpage
The impact of these grass roots movements is seen in Open Data (dot) gov. Pioneered by leaders such as Tim Berners Lee and Nigel Shadbolt
The Shakespeare review [-@shakespeare_shakespeare_2013] indicate that the amount of government Open Data, at least in the UK, is only going to grow. Open data has the potential to trigger a revolution in how governments think about providing services to citizens and how they measure their success: this produces societal impact. This will require an understanding of citizen needs, behaviours, and mental models, and how to use data to improve services.
\newpage
A McKinsey Global Institute report examines the economic impact of Open Data [@mckinsey_open_2013] and estimates that globally open data could be worth a minimum of $3 trillion annually.
\newpage
Open inquiry is at the heart of the scientific enterprise..... Science’s powerful capacity for self-correction comes from this openness to scrutiny and challenge.
Science as an open enterprise [@royal_society_science_2012 p. 7].
Science is based on building on, reusing and openly criticising the published body of scientific knowledge.
For science to effectively function, and for society to reap the full benefits from scientific endeavours, it is crucial that science data be made open.
The Panton Principles (@murray-rust_panton_2010) which underpin Open Science.
The Royal Society’s report Science as an open enterprise [-@royal_society_science_2012] identifies how 21^st^ century communication technologies are changing the ways in which scientists conduct, and society engages with, science. The report recognises that ‘open’ enquiry is pivotal for the success of science, both in research and in society.
The Panton Principles pre-cursed this call with a clarion call to the academic community to open their data and start to conduct open science.
This goes beyond open access to publications (Open Access), to include access to data and other research outputs (Open Data), and the process by which data is turned into knowledge (Open Science).
Zenodo is a DATA REPOSITORY which offers:
\newpage
\newpage
In [21]:
from IPython.display import IFrame
IFrame('http://www.osgeo.org/', width=1200, height=700)
Out[21]:
\newpage
\newpage
The Defense domain are a bit more explicit......
As defined by DoD policy, interoperability is the ability of systems, units, or forces to provide data, information, material, and services to, and accept the same from, other systems, units, or forces; and to use the data, information, material, and services so exchanged to enable them to operate effectively together. IT and NSS interoperability includes both the technical exchange of information and the end-to-end operational effectiveness of that exchanged information as required for mission accomplishment. Interoperability is more than just information exchange; it includes systems, processes, procedures, organizations, and missions over the life cycle and must be balanced with information assurance.
@watson_joint_2010
\newpage
\newpage
\newpage
\newpage
\newpage
See @innocenti_towards_2011 for more details
\newpage
Social interoperability is concerned about the environment and business and human processes.
US Department of Defence researchers have advocated the development of Policy, Standards, and Operational Procedures for:
\newpage
Legal interoperability addresses the process of making legal rules cooperate across jurisdictions, on different subsidiary levels within a single state or between two or more states.
(@weber_legal_2014, p. 6)
The Research Data Alliance state that legal interoperability occurs among multiple datasets when:
Legal interoperability also implies that the search for or tracking of licenses or other legal instruments and their compatibility with other legal conditions will occur in online environments.
\newpage
\newpage
There is a multitude of formal and informal data.
\newpage
Each of these data objects can be licenced in a different way. This shows some of the licences described by the RDFLicence ontology
\newpage
In [ ]:
### Export this notebook as markdown
commandLineSyntax = 'dot -Tpng FCA_ConceptAnalysis.dot > FCA_ConceptAnalysis.png'
commandLineSyntax = 'dot -Tsvg FCA_ConceptAnalysis.dot > FCA_ConceptAnalysis.svg'
print (commandLineSyntax)
os.system(commandLineSyntax)
Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:
Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.
I'll describe CC in more detail
\newpage
\newpage
The following clauses constrain CC0:
\newpage
\newpage
Certain licences when combined:
\newpage
\newpage
A conceptual licence processing workflow. The licence processing service analyses the incoming licence metadata and determines if the data can be legally integrated and any resulting licence implications for the derived product.
\newpage
Data1 hasDerivedContentIn NewThing.
Data1 hasLicence a cc-by-sa.
What hasLicence a cc-by-sa? #reason here
If X hasDerivedContentIn Y and hasLicence Z then Y hasLicence Z. #reason here
Data2 hasDerivedContentIn NewThing.
Data2 hasLicence a cc-by-nc-sa.
What hasLicence a cc-by-nc-sa? #reason here
Nothing hasLicence a cc-by-nc-sa and hasLicence a cc-by-sa. #reason hereAnd processing this within the Protege reasoning environment
In [ ]:
from IPython.display import YouTubeVideo
YouTubeVideo('jUzGF401vLc')
\newpage
In [12]:
from IPython.display import YouTubeVideo
YouTubeVideo('tkRB5Rp1_W4')
Out[12]:
\newpage
| License | Reproduction | Distribution | Derivation | BY | SA | NC |
|---|---|---|---|---|---|---|
| CC0 | X | X | X | |||
| CC-BY-ND | X | X | X | |||
| CC-BY-NC-ND | X | X | X | X | ||
| CC-BY | X | X | X | X | ||
| CC-BY-SA | X | X | X | X | X | |
| CC-BY-NC | X | X | X | X | X | |
| CC-BY-NC-SA | X | X | X | X | X | X |
| ODC-PDDL | X | X | X | |||
| ODC-BY | X | X | X | X | ||
| ODC-ODbL | X | X | X | X | X | |
| OGL 2.0 | X | X | X | X | ||
| OS OpenData | X | X | X | X | ? |
\newpage
\newpage
\newpage
\newpage
\newpage
RDF stores data as triples in the following manner:
This is a graph model that consists of nodes (subject and object)) and edges (predicate).
\newpage
\newpage
\newpage
RDF can be represented in different ways - each of which are interoperable. For example:
Each represent subject, predicate, object triples in different ways
\newpage
★ Available on the web (whatever format) but with an open licence, to be Open Data
★★ Available as machine-readable structured data (e.g. excel instead of image scan of a table)
★★★ as (2) plus non-proprietary format (e.g. CSV instead of excel)
★★★★ All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff
★★★★★ All the above, plus: Link your data to other people’s data to provide context\newpage
\newpage
In [14]:
from IPython.display import IFrame
IFrame('http://lov.okfn.org/dataset/lov/', width=1000, height=700)
Out[14]:
\newpage
\newpage
An ontology is a shared formal explicit specialisation of a conceptualisation
After Agarwal -(@agarwal_ontological_2005):
\newpage
We can use these concepts to infer new information from facts.
For example: from the fact 'Ching Ching' is a newborn Panda we know:
'Ching Ching' is a Panda.
'Ching Ching' is a newborn.We can infer:
'Ching Ching' is a Bear.
'Ching Ching' is a Carnivore. ????
'Ching Ching' eats only meat. ????If we had other logic that told us that 'newborn' is the same as saying less than one year then we can also infer
'Ching Ching' is a Cub.In an ontology/RDF you can say Anything about Anything. Whilst carnivore is a generally useful concept about bears it is not specifically useful when considering pandas. The domain of application is clearly important.
\newpage
Find me the capital of all countries in Africa:
PREFIX abc: <nul://sparql/exampleOntology#> .
SELECT ?capital ?country
WHERE {
?x abc:cityname ?capital ;
abc:isCapitalOf ?y.
?y abc:countryname ?country ;
abc:isInContinent abc:Africa.
}There is a thing ('x') against which the following concepts exist:
The 'concept for which the city is capital' (stored in variable 'y') must also have the following concepts:
\newpage
An OGC standard
SELECT ?f
WHERE { ?f my:hasPointGeometry ?fGeom .
?fGeom ogc:asWKT ?fWKT .
FILTER (ogcf:relate(?fWKT,
“<http://www.opengis.net/def/crs/OGC/1.3/CRS84>
Polygon ((-83.5 34.0, -83.5 34.3, -83.1 34.3,
-83.1 34.0, -83.5 34.0))”^^ogc:WKTLiteral,
ogc:within))
}
In [19]:
from IPython.display import IFrame
IFrame('http://www.opengeospatial.org/projects/groups/geosparqlswg', width=1000, height=700)
Out[19]:
\newpage
\newpage
In [22]:
from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/doc/50kGazetteer/177276', width=1000, height=700)
Out[22]:
\newpage
In [24]:
from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/id/postcodeunit/NG72QL', width=1000, height=700)
Out[24]:
\newpage
In [20]:
from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/', width=1000, height=700)
Out[20]:
\newpage
In [21]:
from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/ontology/', width=1000, height=700)
Out[21]:
\newpage
In [17]:
from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/datasets/code-point-open/explorer/sparql', width=1000, height=700)
Out[17]:
\newpage
In [25]:
from IPython.display import IFrame
IFrame('http://linkedgeodata.org/About', width=1000, height=700)
Out[25]:
\newpage
In [26]:
from IPython.display import IFrame
IFrame('http://browser.linkedgeodata.org/', width=1000, height=700)
Out[26]:
\newpage
In [27]:
from IPython.display import IFrame
IFrame('http://www.geonames.org/ontology/documentation.html', width=1000, height=700)
Out[27]:
\newpage
In [28]:
from IPython.display import IFrame
IFrame('http://www.geonames.org/maps/google_52.94_358.8.html', width=1000, height=700)
Out[28]:
\newpage
In [29]:
from IPython.display import IFrame
IFrame('http://lov.okfn.org/dataset/lov/vocabs/gn', width=1000, height=700)
Out[29]:
\newpage
In [30]:
from IPython.display import IFrame
IFrame('http://lov.okfn.org/dataset/lov/vocabs/?q=geo+space+address+geonames+os+spatial', width=1000, height=700)
Out[30]:
\newpage
\newpage
Geo example:
Leeds is a city.
Yorkshire is a county.
Sheffield is a city.
Lancaster is a city.
Lancashire is a county.
Lancaster has a port.
What is Leeds?
Leeds isIn Yorkshire.
Sheffield isIn Yorkshire.
Lancaster isIn Lancashire.
What isIn Yorkshire?
If X isIn Y then Y contains X.
What contains Leeds?
Yorkshire borders Lancashire.
If X borders Y then Y borders X.
What borders Lancashire?
Yorkshire isIn UnitedKingdom.
Lancashire isIn UnitedKingdom.
#Transitivity
If X isIn Y and Y isIn Z then X isIn Z.
If X contains Y and Y contains Z then X contains Zusing proper isIn
Leeds is a city.
Yorkshire is a county.
Sheffield is a city.
Lancaster is a city.
Lancashire is a county.
Lancaster has a port.
What is Leeds?
Leeds is spatiallyWithin Yorkshire.
Sheffield is spatiallyWithin Yorkshire.
Lancaster is spatiallyWithin Lancashire.
What is spatiallyWithin Yorkshire?
If X is spatiallyWithin Y then Y spatiallyContains X.
What spatiallyContains Leeds?
Yorkshire borders Lancashire.
If X borders Y then Y borders X.
What borders Lancashire?
Yorkshire is spatiallyWithin UnitedKingdom.
Lancashire is spatiallyWithin UnitedKingdom.
#Transitivity
If X is spatiallyWithin Y and Y is spatiallyWithin Z then X is spatiallyWithin Z.
If X spatiallyContains Y and Y spatiallyContains Z then X spatiallyContains Z
What is spatiallyWithin UnitedKingdom?Adding more......
Pudsey is spatiallyWithin Leeds.
Kirkstall is spatiallyWithin Leeds.
Meanwood is spatiallyWithin Leeds.
Roundhay is spatiallyWithin Leeds.
Scarcroft is spatiallyWithin Leeds.and more
UnitedKingdom isPartOf Europe.
UnitedKingdom is a country.
If X isPartOf Y and X spatiallyContains Z then Z isPartOf Y.
What isPartOf Europe?
In [ ]:
and more
```
If X spatiallyContains Y and X is a city then Y is a place and Y is a cityPart.
Every city is a place.
What is a place.
```
In [ ]:
and more
```
UK isPartOf Europe.
UK is sameAs UnitedKingdom.
If X has a port then X borders Water.
What borders Water?
```
\newpage