This notebook covers the setup process for the tutorials that follow. The setup is then tested with a simple example
To work through these tutorials, you will need
The later tutorials use some additional components:
I'll assume that you have eWater Source installed. Everything else is described, below.
You should be able to install and run everything without administatrive privileges. However, if you want to run examples where Source is running remotely, on a different machine to Python, then you will need admin privileges to register the Veneer service on the machine running Source.
It is easy to install standard Python, but the 'official' distribution of Python doesn't include the libraries/packages typically used in scientific computing and data analysis environments.
Anaconda Python is a third party distribution of Python that specific targets science and data analysis users. It includes most of additional packages that we'll need (except veneer-py of course!). Using Anaconda saves a good deal of setup work. Anaconda doesn't include the key geospatial packages, but they can be installed as needed.
Install Anaconda from https://www.continuum.io/downloads. Download the version that relates to Python 3 (Currently Python 3.5). Python is currently at major version 3, but the incompatible Python 2 is still popular. For this reason, Anaconda currently separate distributions for Python 2 (2.7) and Python 3 (3.5). These tutorials, and the veneer-py package, are tested with Python 3.5.
You should be able to accept the defaults when installing Anaconda. Note If you have another Python environment (eg as part of ArcGIS) consider carefully whether Anaconda should register itself as the default Python.
One of the main capabilities that Anaconda installs is the Jupyter Notebook system (still referred to as the IPython Notebook in some material). This is a browser based environment for creating and running scripts, capturing documentation and visualising outputs.
Veneer is a Source plugin (ie a DLL). The latest release is available at https://github.com/flowmatters/veneer/releases. You need to download the version appropriate to your version of Source. (At the time of writing, the latest versions of Veneer work with 3.8.22 and up. Older versions of Veneer are available for older versions of Source, but the old versions miss certain features needed for these tutorials.)
Notes:
From https://github.com/flowmatters/veneer/releases, download the Veneer release for your version of Source. Unzip it to a convenient location.
If you only plan to use Veneer from the main Windows application version of Source, you can place the unzipped files anywhere. However, if you plan to use the command line version of Veneer, it will require access to all of Source's other DLL and EXE files. In this case, you can either place the unzipped Veneer into the main Source folder (eg C:\Program Files\eWater\Source 4.1.0.YYYY), OR copy all of the files from the main Source folder to the location where you unzipped Veneer.
FlowMatters.Source.Veneer.dll.
You need to load a Source project before you can start Veneer:
Started Source RESTful Service on port: 9876 then Veneer is working. You can confirm it is working by going to your web browser by visiting the following url: http://localhost:9876/network.Requested network. Your browser is probably showing a jumble of text (in JSON format). This represents the node link network of your current Source model. Once we've installed veneer-py, we'll request the network from Python and see that Python can interpret the data and let us further query the network.A few things can go wrong at this point.
FlowMatters.Source.Veneer.dll), but you may need to do so manually for the other DLLs in the Veneer directory:
COULD NOT START VENEER.... This indicates a permissions issue is blocking you from starting Veneer. Veneer communicates to other software over network ports - even when working locally - and it does this by running a server on a designated port. Now, if another program is already using the default port (9876), then Veneer will try the next port (9877) and so on until a free port is available. This means if you start two copies of Source concurrently and use Veneer in both, they will register themselves on different ports. Veneer will report an error if it is blocked from using a port for some reason other than the port being in use. The most common reason to get this error at start up is if you have previously registered to allow remote connections on this port and you subsequently try to run a local-only connection (see below). If this is the case, you should be able to check the Allow Remote Connections box then click the Start button to try starting the server.The Web Server Monitoring window provides control to Start, Stop and Restart the server, and to Clear the log messages. There are also three options that control how Veneer operates:
veneer-py is a Python package. The pip tool, which was installed with Anaconda, can be used to download and install veneer-py.
You can run pip from a Windows Command Prompt. Note, if you have more than one Python environment installed (eg because you use Python with ArcGIS), then you should use the Anaconda Command Prompt, which was installed with Anaconda. From the command prompt, run the folllowing command
pip install https://github.com/flowmatters/veneer-py/archive/master.zipWith veneer-py installed, you can now use it, from Python, to communicate with Source and Veneer.
veneer-py is regularly updated. At this stage, there is no explicit version number - only the latest revision of the master branch of the git repository. To upgrade to the latest version, you first uninstall veneer-py then install the latest:
pip uninstall veneer-py
pip install https://github.com/flowmatters/veneer-py/archive/master.zip
In [ ]:
There are a number of Python packages for spatial analysis and these can be very useful when using veneer-py - particularly with spatial models.
Historically, installing some of these packages on Windows has been tricky and a number of non-standard approaches have emerged.
condaOne option, that worked for many people, uses the conda tool, supplied with Anaconda:
conda install -c conda-forge shapely
conda install -c conda-forge fiona
conda install -c conda-forge rasterio
conda install geopandas
pip install rasterstatsThis will install the named packages (fiona, shapely, rasterio, geopandas, rasterstats) as well as a number of notable dependencies, including gdal.
.whl files):There is a repository of prebuilt scientific Python packages for Windows, maintained by Christoph Gohlke of Universiry of California, Irvine at http://www.lfd.uci.edu/~gohlke/pythonlibs/. For users that are unsuccessful with the conda installation, using these packages should work.
Shapely-1.5.17-cp36-cp36m-win_amd64.whlpip install <fn> for each .whl file downloadedpip install geopandas rasterstats to install remaining packagesThe Jupyter Notebook runs as a local server application, hosting notebooks - interactive documents featuring code, data and documentation. Like Veneer, the Notebook runs on http, which, by default, is only available from your local machine.
Anaconda installs Jupyter Notebook in your start menu. You can use this option, but it will, by default, start the notebook server in your Windows home directory - and you will only be able to load notebooks store within this directory (including subdirectories). If you want to store notebooks in other directories, the notebook server needs to be started in those directories. You can modify the shortcut installed by Anaconda, but I typically just start the notebook server from the Windows Command Prompt (or the Anaconda Command Prompt). For example:
cd \work\source_projects
jupyter notebookRegardless of whether you use the start menu or the command prompt, the server will start and then a new tab will open in your we browser, pointing to the notebook server (on port 8888 by default). If you close that tab (or want to use a different browser), the default notebook server should be at http://localhost:8888. However, like Veneer, if the notebook server finds its default port occupied, it will increment the port number, to 8889 and try again, and so on.
Note: The Jupyter Notebook is sometimes still referred to as the IPython Notebook. IPython was the original project, Jupyter is the evolution of that project into a multi-language notebook environment.
In [ ]:
When working with Veneer and veneer-py, its common to create a number of notebooks. Some will be short experiments, others will capture important logic that you want to be able to repeat. Depending on your needs, notebooks may be closely coupled to a particular Source model, or the notebooks may be more generic and work with a range of Source models.
Personally, most of my notebooks relate to a single Source model and as a result, I keep the notebooks in the same directory as the model file (.rsproj). This way, I can easily find the notebooks that I've used with a particular model, and the
In [ ]:
Notebooks are interactive documents.
The documents are made of cells. This cell is a 'Markdown' cell - used for documentation.
Code cells hold source code - Python in these tutorials. You can run a code cell to execute the code and to see any results.
There are numerous tutorials available for the notebook (though many focus on particular domains). If you are reading these instructions in a notebook server, you can run the following two code cells using the notebook controls (eg the Run button on the toolbar). Alternatively, start a new notebook and past the text from the code cell into a code cell in the notebook, then run.
If you haven't yet installed Anaconda or started the notebook server, you can try it out online at https://try.jupyter.org/.
In [7]:
#### This is a CODE cell.
#### To Run, click on the cell then click the Play button in the toolbar, or the menu Cell|Run, or press Shift+Enter
%matplotlib notebook
import pandas as pd
import numpy as np
import matplotlib
from matplotlib import pyplot as plt
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts = ts.cumsum()
df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
df.plot(); plt.legend(loc='best')
Out[7]:
In [8]:
### This is also a CODE cell,
### Run to see a result
m = 50
a = 9.8
F = m*a
F
Out[8]:
In [ ]:
In [ ]:
In [12]:
import veneer
If there is no error after an import statement, you can assume that it worked - it imported something of that name!
Now, create a Veneer client to talk to the server running within Source. (So, you'll need to have Source running with the Web Server Monitoring open)
In [13]:
v = veneer.Veneer()
You can now use the v symbol to refer to the connection to Source/Veneer. This will be connecting on the default port of 9876. If the web server is running on a different port, you can specify it when setting up the client:
v = veneer.Veneer(port=9877)Now, you can ask the client to interact with Source. We'll test everything is working by querying the network (as we did with the browser)
In [14]:
network = v.network()
You don't see the data that's returned, but you can query it...
In [15]:
all_nodes = network['features'].find_by_feature_type('node')
In [16]:
all_nodes._unique_values('name')
Out[16]:
In [ ]:
The following tutorials are focussed on Veneer and veneer-py - the tutorials are not intended as training in Python or Python Data Analysis.
My favourite book on data analysis in Python is Python for Data Analysis, by Wes McKinney. It's now a few years old, but its still quite relevant for the key concepts and packages (pandas, numpy and the notebook).
If you are familiar with R, there are various resources to support the migration from R to Python, including this blog post.
The Python Data Analysis Library (pandas) is probably the key library you'll use with veneer-py. There are a number of tutorials for pandas at http://pandas.pydata.org/pandas-docs/version/0.18.1/tutorials.html.
There are a great many tutorials and books on Python itself. Think Python is a freely available e-book written from a science perspective.
In [ ]: