In [ ]:
If you haven't already started Source, do so now, following the instructions in 0Setup_and_Hello_World.ipynb to load and start Veneer using the Tools|Web Server Monitoring option.
Note: This session uses ExampleProject/RiverModel1.rsproj. You are welcome to work with your own model instead, however you will need to change the notebook text at certain points to reflect the names of nodes, links and functions in your model file.
You can edit your local copy of this notebook. Alternatively, start a new notebook and enter the relevant code there. From an existing notebook, use the File|New Notebook|Python 3 menu to start a new notebook using Python 3 syntax.
You can also create a new notebook from the main Jupyter dashboard, using the New button.
It's probably worth using a Markdown cell at the top of the notebook to describe the purpose of the notebook (if only to remind yourself later...)
Then, in the first code block, you should import relevant Python packages and generally initialise your Python environment with anything needed for the notebook:
In [1]:
import veneer
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
The Python import statement makes other packages available to your current session.
There are a few forms of import, two of which are illustrated here:
import veneer makes the main veneer-py package available. You can access parts of the package using veneer.<member>import pandas as pd imports the pandas package, but aliases it as pd for convenience. For common packages (such as numpy adn pandas) there is often a well-used abbreviation, such as np and pd.from os import mkdir, makedirs imports one or more specific members of the os package and makes them available without any prefixfrom os import * imports all the members of the os package and makes them available without any prefix (eg makedirs('test')).%matplotlib enables plotting by configuring a rendering engine for the matplotlib package. Anything starting with % is a special ('magic') command to Jupyter/IPython - it isn't standard Python.
The %matplotlib magic command accepts an optional parameter indicating which rendering backend to use for the current session. I've used inline, which renders charts directly within the notebook as HTML <img> tags. I find this best for creating self-sufficient notebooks that can be circulated and read without running. The downside is that the charts aren't interactive.
Other options worth noting:
%matplotlib - (ie without specifying a rendering backend). This gives you the default for your current system. On my Windows machine, this uses the Qt4 windowing toolkit and charts appear in their own window, with basic zoom, pan, export and layout controls.%matplotlib notebook - Renders within the notebook with some basic interactive chart controls.
In [2]:
%matplotlib inline
plt.plot(range(10))
Out[2]:
In [3]:
%matplotlib notebook
In [4]:
plt.plot(range(10))
Out[4]:
In [5]:
%matplotlib inline
Aside: the range function is a generator. In this case, its generating a consecutive integers, starting from 0 and going up to, but not including, 10.
Try:
print(range(10))
and
print(list(range(10))
In [ ]:
veneer-py provides functions for accessing a running copy of Source/Veneer. Most of these functions are provided through an instance of the Veneer client class, within the veneer package. You can have as many clients as you need - for example, to point to different copies of Source that are running concurrently.
By default, the client will attempt to connect to a copy of Source/Veneer running on the local machine and hosted on port 9876
In [6]:
v = veneer.Veneer()
# Equiavelent to
# v = veneer.Veneer(host='localhost',port=9876)
In [ ]:
The rest of this tutorial uses v to refer to current Veneer client
veneer-py provides two groups of functions:
v.network() accesses the /network URL), andv.model.catchments.runoff.get_param_values('SMSC') uses IronPython to query the SMSC parameter in the rainfall runoff models)The functions in group 1 should work on all Source models, whereas the functions in group 2 tend to be more model specific.
This section deals with group 1 - functions related to specific URLs.
In [7]:
the_network = v.network()
In [8]:
all_nodes = the_network['features'].find_by_feature_type('node')
len(all_nodes)
Out[8]:
In [9]:
all_nodes._all_values('name')
Out[9]:
You can also perform some basic topological queries on the network, such as finding all network outlets, or finding upstream/downstream links for a given node
In [10]:
outlets = the_network.outlet_nodes()
outlets
Out[10]:
In [11]:
the_network.upstream_links(outlets[0])
Out[11]:
If you have GeoPandas installed, you can convert network to a GeoDataFrame, which is useful for visualisation, reporting and advanced filtering.
In [12]:
network_df = the_network.as_dataframe()
network_df
Out[12]:
With a GeoDataFrame, you can filter against any column
In [13]:
network_df[network_df.feature_type=='node']
Out[13]:
In [14]:
network_df[network_df.icon=='/resources/StorageNodeModel']
Out[14]:
In [15]:
model_functions = v.functions()
model_functions
Out[15]:
In [16]:
model_variables = v.variables()
model_variables
Out[16]:
Both v.functions() and v.variables() return a searchable list:
In [17]:
model_variables.find_by_Name('$Runoff')
Out[17]:
Note: When you get a searchable list, you can also convert it to a Pandas Data Frame if that is more convenient
In [18]:
model_variables.as_dataframe()
Out[18]:
v.variables() only returns summary information about the variable. If you want to get at the actual time series (or the piecewise linear function or similar), use v.variable_time_series(name) (or v.variable_piecewise(name)):
In [19]:
runoff_ts = v.variable_time_series('$Runoff')
runoff_ts.plot()
Out[19]:
In [20]:
mfr_piecewise = v.variable_piecewise('MFR_Piecewise')
mfr_piecewise
Out[20]:
Note: At this stage, only Time Series and Piecewise Linear variables are supported through this mechanism
In [21]:
data_sources = v.data_sources()
data_sources
Out[21]:
Using .as_dataframe() can help to interpret the information:
In [22]:
data_sources.as_dataframe()
Out[22]:
This highlights that the Items column has nested information:
In [23]:
data_sources[0]['Items']
Out[23]:
So, v.data_sources() returns information about all data sources. Specifically, it returns the list of data sources, and, for each one, information about the different time series. However, v.data_sources does not return the entire time series. (This is mainly for speed - on big models, this can be HUGE)
To get to the time series, either use v.data_source(name) to retrieve a single data source (and all the time series within it) OR v.data_source_item(source,name) to retrieve a particular time series.
Note: With v.data_source_item you can retrieve the time series for an item that is associated with a particular input set.
In [24]:
forcing = v.data_source('ForcingData_csv')
forcing
Out[24]:
You can see that the time series are included, but they are still nested within the returned data. Essentially, Veneer has returned the data for each input set, with each being a separate Pandas DataFrame.
You can get to an individual DataFrame back with find_one_by_Name
In [25]:
forcing_as_df = forcing['Items'].find_one_by_Name('Default Input Set')['Details']
forcing_as_df[0:30] # Look at the first 30 time steps
Out[25]:
You can jump to a particular data source item (typically a column from an original file) with v.data_source_item(source,name)
In [26]:
rainfall = v.data_source_item('ForcingData_csv','Rainfall')
rainfall[0:30]
Out[26]:
In [ ]:
In [27]:
input_sets = v.input_sets()
input_sets
Out[27]:
Within a given input set, the individual commands are available in the Configuration property
In [28]:
input_sets.find_one_by_Name('Default Input Set')['Configuration']
Out[28]:
Input sets are a lot more interesting when you are changing them and applying them from Python...
In [29]:
response,run_url = v.run_model()
response,run_url
Out[29]:
After a successful run, veneer-py returns the URL of the new run - this can be used to retrieve an index of all the available results and, from there, the actual time series results
In [30]:
run_url
Out[30]:
In [31]:
index = v.retrieve_run(run_url)
index
Out[31]:
This index lists all of the time series (under ['Results']) and can be used to retrieve individual time series, or collections of time series
In [32]:
index['Results'].as_dataframe()
Out[32]:
The easiest way to retrieve time series results is with v.retrieve_multiple_time_series, which takes a set of run results, some search criteria by which it identifies wanted time series. There are also options for retrieve aggregated data and the ability to rename time series as they are retrieved.
The help text for retrieve_multiple_time_series describes the options
In [33]:
help(v.retrieve_multiple_time_series)
The following examples retrieve data from the latest run, using the index index we saved, earlier:
In [34]:
# Retrieve Downstream Flow Volume, anywhere it is recorded.
# Name the columns for the network location
ds_flow = v.retrieve_multiple_time_series(run_data=index,criteria={'RecordingVariable':'Downstream Flow Volume'},
name_fn=veneer.name_for_location)
ds_flow[0:10]
Out[34]:
In [35]:
# Retrieve everything available from the Water User node
# Name the columns for the variable
water_user = v.retrieve_multiple_time_series(run_data=index,criteria={'NetworkElement':'Water User'},
name_fn=veneer.name_for_variable)
water_user[0:10]
Out[35]:
There are numerous functions and options that are closely related to running simulations:
v.run_model() with alternate start and end dates, along with other options related to the simulation, such as specifying an input setv.apply_input_set() to run the commands in a particular input set. This doesn't trigger a run, so its useful in situations where you want to run multiple input sets before a single simulationv.configure_recording() can switch time series recorders on and offv.drop_run) or drop all runs (v.drop_all_runs())These actions will all be covered in subsequent tutorials. For now, a simple example of enabling recording of Storage Surface Area and rerunning:
In [36]:
v.drop_all_runs()
v.configure_recording(enable=[{'RecordingVariable':'Storage Surface Area'}])
v.run_model()
Out[36]:
In [37]:
surface_area = v.retrieve_multiple_time_series(criteria={'RecordingVariable':'Storage Surface Area'})
surface_area.plot()
Out[37]:
Note: We again drop all runs before moving on...
In [38]:
v.drop_all_runs()
In [ ]:
There are various ways to use veneer-py to change the configuration of the currently loaded model in Source
v.update_input_set)v.apply_input_set)v.model. to directly change model parameters and structureIt is also possible to modify files on disk where Source is configured to 'Reload-On-Run', such as changing time series files or input set command files. This option isn't specific to Veneer/veneer-py and isn't covered in these tutorials.
We can update the input set commands for the default input set.
(Making a single change like this can be a bit fiddly!)
In [39]:
default_input_set = v.input_sets()[0]
default_input_set
Out[39]:
In [40]:
command_of_interest = default_input_set['Configuration'][0]
command_of_interest
Out[40]:
In [41]:
default_input_set['Configuration'][0] = command_of_interest.replace('25 m','50 m')
In [42]:
default_input_set
Out[42]:
In [43]:
v.update_input_set('Default Input Set',default_input_set)
Out[43]:
At this point, you can check in the Source user interface (Edit|Scenario Input Sets) to see that the input set has been updated. Alternatively, you can query the input set again
In [44]:
v.input_sets()
Out[44]:
The Default Input Set would be applied when we next run, but if we want those parameters set on the model immediately, we can do so, with v.apply_input_set('Default Input Set')
In [45]:
v.apply_input_set('Default Input Set')
Out[45]:
v.model and the Allow Scripts optionThe v.model namespace contains a lot of functionality for modifying the structure of the Source model. This part of veneer-py works by generating IronPython scripts, which run directly within Source and manipulate the .NET objects that Source is built upon.
This capability (to accept IronPython scripts) is disabled, by default, in Veneer. To enable it, click the 'Allow Scripts' checkbox in the Web Server Monitoring window. If you don't enable the 'Allow Scripts' option, then the following instructions will give an error.
As an example of what can be done, the following command queries the routing model used for each link
In [46]:
v.model.link.routing.get_models()
Out[46]:
This shows that, for the example model, there are three Straight Through Routing links and two Storage Routing links.
get_models() has returned a list - and this is common within the v.model namespace. It's also common for functions to accept a list when changing the Source model (eg by changing assigned models, or changing parameters).
The list isn't directly useful though, because it doesn't tell you which link is which. That can be checked by asking for the results coded by name
In [47]:
v.model.link.routing.get_models(by_name=True)
Out[47]:
In [ ]:
PEST is a powerful, model independent Parameter ESTimation tool. veneer-py has some initial functionality for configuring and running PEST jobs over Source models using Veneer.
This functionality is currently the best demonstration of parallel processing using Veneer and veneer-py. This is explored in Tutorial 8.
In [ ]:
Veneer is distributed with a command line program, FlowMatters.Source.VeneerCmd.exe, which is an alternative to the eWater supplied RiverSystem.CommandLine.exe. The Veneer command line uses the same protocols as the plugin version of Veneer, making them interchangeable for most applications.
Note: To use the Veneer command line, it needs access to all of the DLLs distributed with Source. Currently, this means that the Veneer command line, and the various DLLs related to Veneer, need to be in the same directory as these eWater supplied files. You can copy all of the eWater files into the Veneer directory, or copy all the Veneer files into the Source installation directory (typically under C:\Program Files\eWater\Source X.Y.ZZ.WWWW). Importantly, if copying the Veneer files to the Source directory, the Veneer files must be in the same directory as the main Source files (eg RiverSystem.Forms.exe), not in the plugins directory
In [48]:
from veneer.manage import start
In [49]:
help(start)
Each copy of the Veneer command line is a copy of the Source engine with a Veneer server.
Each running process will run until terminated, either by the user or by the end of the Python session.
Veneer communicates via network ports, but by default, these connections are restricted to local machine - ie you cannot connect to Veneer on machine A from a Python notebook running on machine B.
If you want to allow remote connections, you need to enable the checkbox in the Web Server Monitoring window. However, in order to do so, you will first need to register the Veneer URL with Windows. Instructions are at https://github.com/flowmatters/veneer.
In [ ]:
In [50]:
help(v.network)
<tab> completion is one of the nice features of Jupyter.
In this way, you can navigate veneer-py (and other Python packages) to discover relevant functions:
v.model.<press tab>
In [ ]:
In [ ]: