In [ ]:
This workshop is all about automating interactions with eWater Source, using Veneer (an alternative to the RiverSystem.CommandLine tool) and veneer-py (a Python package that provides a high level interface to Veneer and Source).
Some of the tutorials will focus on running Source (much as you could with the existing command line tool) and then post processing the results. Other tutorials will focus on making changes to the model structure that are difficult to perform through the command line.
I hope that, after completing the tutorials, you will:
Each of the tutorials will expect you to have Source running, with a project loaded. The Veneer plugin should be installed (as per instructions in Session 0) and the Web Server Monitoring tool should be started.
Example model files are provided for each tutorial, although you are welcome to use a model of your own. If you use you're own project, you will need to modify some of the example code in the tutorials to match the structure of your model. For example, you may need to change node names or function names.
Look for the Which Model section in each tutorial to see which model has been used in the design of the tutorial. The example models have all been built to run in the Source 4.1.1 Public Release. The examples are found in the ExampleProject directory.
In this session, you can use any of the example models (RiverModel1.rsproj, RiverModel2.rsproj or CalibrationExample.rsproj). Any Source model you have would also work equally well in this session.
Our interaction with Source is handled through the Veneer plugin to the Source application.
Veneer allows other programs to talk to Source,
Veneer can work within the regular Windows application version of Source. Veneer also has its own command line, that accepts the same protocols and can be used for larger scale tasks, such as parallel calibration.
Veneer is open source. The code is hosted at https://github.com/flowmatters/veneer. You can download the most recent Veneer release from there as well - there will be downloads for each version of Source that is currently supported with Veneer.
Veneer works by turning Source into a server, which can accept instructions from other programs.
This is analogous the server mode of RiverSystem.CommandLine, but where the command line tool is just on the command line, Veneer can work in either a command line mode or from the main Windows application of Source.
Veneer and RiverSystem.CommandLine exist for different reasons. RiverSystem.CommandLine was built primarily to support automated running of Source simulations, wheareas the original purpose of Veneer was to support the development of web based user interfaces for Source.
This leads to some design differences between the two:
RiverSystem.CommandLine is focussed primarily on running simulations and retrieving results, whereas Veneer also exposes a number of model configuration elements, such as the networkRiverSystem.CommandLine uses XPath, whereas Veneer uses IronPython scripts.Veneer establishes a server - so other programs (whether they be Python scripts, web applications or something else) become clients of the server.
Veneer communicates with its clients using HTTP.
Clients initiate communication by requesting a particular URL on the Veneer server, specifying a HTTP verb (or action) and, in some cases, supplying some additional data, such as model parameters.
Veneer responds to these requests by taking relevant action on the Source model (querying a result, running the model, etc) and returning a response to the client, typically as JSON formatted text.
Because Veneer uses URLs, many of the queries can be tested by visiting those URLs in a browser:
Note: in the case of time series, the request can specify that the response should be in CSV format (rather than JSON), but this typically isn't necessary when calling from Python. For web applications, the more compressed nature of CSV data can speed up application response times.
These queries all work in your browser because Veneer is expecting HTTP GET requests - which is what your browser issues when you enter a URL in the address bar. Other Veneer actions, such as triggering a run, using HTTP the POST, PUT and DELETE verbs, which aren't as easy to test in a browser.
For example, the following Python code issues a POST request to /runs in Veneer, triggering a run of the current scenario in Source
(using the current, default settings).
In [1]:
import requests
requests.post('http://localhost:9876/runs')
Out[1]:
Source is configured by default to have the Results Manager automatically appear after a model run. It makes sense for interactive use, but it can get in the way when running Source from Python.
I recommend switching off the automatic launch of the Results Manager using
Edit | Application Settings
then selecting Results Manager and clearing the Pop up Results Manager after run completion checkbox. Click OK and the results manager will no longer appear automatically after a run. You only need to do this once.
You can always press F10 to open the results manager.
In addition to the various URLs, Veneer also supports a catch-all URL - a POST request to /ironpython - that accepts an IronPython script for execution within the running Source application.
IronPython is a native .NET version of Python, meaning it can directly access the classes and objects within Source. With the /ironpython URL, you can run a script, within Source, that can achieve any kind of query or transformation possible.
The /ironpython URL is disabled by default. You should be careful before enabling it if you are also using the Remote Connections feature.
Writing IronPython scripts for use in Source can be quite complex - it requires a knowledge of the Source data structures and methods relevant to what you want to achieve. However, veneer-py includes a number of helper functions, that can be called from the notebook, which generate scripts for many common tasks, such as assigning input time series, creating nodes, assigning rainfall runoff models and setting model parameters.
Veneer is independent of the client software that talks to it - the can communicate with Veneer with client programs written in any language.
veneer-py is a Python package for simplifying the task of talking to Veneer from Python scripts.
veneer-py has a range of features for making requests to Veneer, and interpreting results.
For example, if you make a request for the network (/network), you don't see the JSON formatted data - you get a Python object that you can query and deal with in a Python way:
In [6]:
import veneer
v = veneer.Veneer() # Start a veneer-py client
network = v.network()
nodes = network['features'].find_by_feature_type('node')
print(len(nodes))
print(nodes._all_values('name')[0:10]) # Names of first ten nodes
In [ ]:
In [ ]: