In [1]:
# Putting the initialisation at the top now!
import veneer
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
v = veneer.Veneer(port=9876)
This session covers functionality in Veneer and veneer-py for making larger changes to model setup, including structural changes.
Using this functionality, it is possible to:
Note: This session uses ExampleProject/RiverModel2.rsproj. You are welcome to work with your own model instead, however you will need to change the notebook text at certain points to reflect the names of nodes, links and functions in your model file.
This is a big topic and the material in this session will only touch on some of the possibilities.
Furthermore, its an evolving area - so while there is general purpose functionality that is quite stable, making the functionality easy to use for particular tasks is a case by case basis that has been tackled on an as-needed basis. There are lots of gaps!
There are various motivations for the type of automation of Source model setup described here. Some of these motivations are more practical to achieve than others!
Could you build a complete Source model from scratch using a script?
In theory, yes you could. However it is not practical at this point in time using Veneer. (Though the idea of building a catchments-style model is more foreseeable than building a complex river model).
For some people, building a model from script would be desirable as it would have some similarities to configuring models in text files as was done with the previous generation of river models. A script would be more powerful though, because it has the ability to bring in adhoc data sources (GIS layers, CSV files, etc) to define the model structure. The scripting approach presented here wouldn't be the most convenient way to describe a model node-by-node, link-by-link - it would be quite cumbersome. However it would be possible to build a domain-specific language for describing models that makes use of the Python scripting.
Most of the practical examples to date have involved applying some change across a model (whether that model is a catchments-style geographic model or a schematic style network). Examples include:
There are several reasons for making changes to the Source model without wanting the changes to be permanently saved in the model.
In [ ]:
This example uses the earlier RiverModel.rsproj example file although it will work with other models.
Here, we will convert all links to use Storage Routing except for links that lead to a water user.
Note: To work through this example (and the others that follow), you will need to ensure the 'Allow Scripts' option is enabled in the Web Server Monitoring window.
In [ ]:
v.model namepsaceMost of our work in this session will involve the v.model namespace. This namespace contains functionality that provides query and modification of the model structure. Everything in v.model relies on the 'Allow Scripts' option.
As with other parts of veneer-py (and Python packages in general), you can use <tab> completion to explore the available functions and the help() function (or the ? suffix) to get help.
In [2]:
existing_models = v.model.link.routing.get_models()
existing_models
Out[2]:
Note:
get_models() functions is available in various places through the v.model namespace. For example, v.model.catchments.runoff.get_models() queries the rainfall runoff models in subcatchments (actually in functional units). There are other such methods, available in multiple places, including:set_modelsget_param_valuesset_param_valuesv.model.link.routing.get_models(links='Default Link #3')
v.model.link.routing.set_models('RiverSystem.Flow.LaggedFlowRoutingWrapper',links='Default Link #3')
v.model.catchment.runoff.get_models(fus='Grazing')
v.model.catchment.runoff.set_models('MyFancyRunoffModel',fus='Grazing')
help(v.model.link.routing)
get_models() return a list of model names. Two observations about this:v.model namespace - it uses the terminology within Source. There are, however, help functions for finding what you need. For example:v.model.find_model_type('gr4')
v.model.find_parameters('RiverSystem.Flow.LaggedFlowRoutingWrapper')
get_ functions return lists (although there is a by_name option being implemented) and the set_ functions accept lists (unless you provide a single value in which case it is applied uniformly). It is up to you to interpret the lists returned by get_* and to provide set_* with a list in the right order. The way to get it right is to separately query for the names of the relevant elements (nodes/links/catchments) and order accordingly. This will be demonstrated!
In [3]:
link_names_order = v.model.link.routing.names()
link_names_order
Out[3]:
OK - that gives us the names - but it doesn't help directly. We could look at the model in Source to work out which one is connected to the Water User - but that's cheating!
More generally, we can ask Veneer for the network and perform a topological query
In [4]:
network = v.network()
Now that we've got the network, we want all the water users.
Now, the information we've been returned regarding the network is in GeoJSON format and is intended for use in visualisation. It doesn't explicitly say 'this is a water user' at any point, but it does tell us indirectly tell us this by telling us about the icon in use:
In [5]:
network['features']._unique_values('icon')
Out[5]:
So, we can find all the water users in the network, by finding all the network features with '/resources/WaterUserNodeModel' as their icon!
In [6]:
water_users = network['features'].find_by_icon('/resources/WaterUserNodeModel')
water_users
Out[6]:
Now, we can query the network for links upstream of each water user.
We'll loop over the water_users list (just one in the sample model)
In [7]:
links_upstream_of_water_users=[]
for water_user in water_users:
links_upstream_of_water_users += network.upstream_links(water_user)
links_upstream_of_water_users
Out[7]:
Just one link (to be expected) in the sample model. Its the name we care about though:
In [8]:
names_of_water_user_links = [link['properties']['name'] for link in links_upstream_of_water_users]
names_of_water_user_links
Out[8]:
To recap, we now have:
existing_models - A list of routing models used on linkslink_names_order - The name of each link, in the same order as for existing_modelsnames_of_water_user_links - The names of links immediately upstream of water users. These links need to stay as Straight Through RoutingWe're ultimately going to call
v.model.link.routing.set_models(new_models,fromList=True)
so we need to construct new_models, which will be a list of model names to assign to links, with the right mix and order of storage routing and straight through. We'll want new_models to be the same length as existing_models so there is one entry per link. (There are cases where you my use set_models or set_param_values with shorter lists. You'll get R-style 'recycling' of values, but its more useful in catchments where you're iterating over catchments AND functional units)
The entries in new_models need to be strings - those long, fully qualified class names from the Source world. We can find them using v.model.find_model_type
In [9]:
v.model.find_model_type('StorageRo')
Out[9]:
In [10]:
v.model.find_model_type('StraightThrough')
Out[10]:
We can construct our list using a list comprehension, this time with a bit of extra conditional logic thrown in
In [11]:
new_models = ['RiverSystem.Flow.StraightThroughRouting' if link_name in names_of_water_user_links
else 'RiverSystem.Flow.StorageRouting'
for link_name in link_names_order]
new_models
Out[11]:
This is a more complex list comprehension than we've used before. It goes like this, reading from the end:
for link_name in link_names_order]
link_name is present in the list of links upstream of water users, use straight through routing['RiverSystem.Flow.StraightThroughRouting' if link_name in names_of_water_user_links
else 'RiverSystem.Flow.StorageRouting'
All that's left is to apply this to the model
In [12]:
v.model.link.routing.set_models(new_models,fromList=True)
Out[12]:
Notes:
fromList parameter tells the set_models function that you want the list to be applied one element at a time.Now that you have Storage Routing used in most links, you can start to parameterise the links from the script.
To do so, you could use an input set, as per the previous session. To change parameters via input sets, you would first need to know the wording to use in the input set commands - and at this stage you need to find that wording in the Source user interface.
Alternatively, you can set the parameters directly using v.model.link.routing.set_param_values, which expects the variable name as used internally by Source. You can query for the parameter names for a particular model, using v.model.find_parameters(model_type) and, if that doesn't work v.model.find_properties(model_type).
We'll start by using find_parameters:
In [13]:
v.model.find_parameters('RiverSystem.Flow.StorageRouting')
Out[13]:
The function v.model.find_parameters, accepts a model type (actually, you can give it a list of model types) and it returns a list of parameters.
This list is determined by the internal code of Source - a parameter will only be returned if it has a [Parameter] tag in the C# code.
From the list above, we see some parameters that we expect to see, but not all of the parameters for a Storage Routing reach. For example, the list of parameters doesn't seem to say how we'd switch from Generic to Piecewise routing mode. This is because the model property in question (IsGeneric) doesn't have a [Property] attribute.
We can find a list of all fields and properties of the model using v.model.find_properties. It's a lot more information, but it can be helpful:
In [14]:
v.model.find_properties('RiverSystem.Flow.StorageRouting')
Out[14]:
Lets apply an initial parameter set to every Storage Routing link by setting:
RoutingConstant to 86400, andRoutingPower to 1We will call set_param_values
In [15]:
help(v.model.link.routing.set_param_values)
In [16]:
v.model.link.routing.set_param_values('RoutingConstant',86400.0)
Out[16]:
In [17]:
v.model.link.routing.set_param_values('RoutingPower',1.0)
Out[17]:
You can check in the Source user interface to see that the parameters have been applied
Often, you will want to calculate model parameters based on some other information, either within the model or from some external data source.
The set_param_values can accept a list of values, where each item in the list is applied, in turn, to the corresponding models - in much the same way that we used the known link order to set the routing type.
The list of values can be computed in your Python script based on any available information. A common use case is to compute catchment or functional unit parameters based on spatial data.
We will demonstrate the list functionality here with a contrived example!
We will set a different value of RoutingPower for each link. We will compute a different value of RoutingPower from 1.0 down to >0, based on the number of storage routing links
In [23]:
number_of_links = len(new_models) - len(names_of_water_user_links)
In [29]:
power_vals = np.arange(1.0,0.0,-1.0/number_of_links)
power_vals
Out[29]:
In [30]:
v.model.link.routing.set_param_values('RoutingPower',power_vals,fromList=True)
Out[30]:
If you open the Feature Table for storage routing, you'll now see these values propagated.
The fromList option has another characteristic that can be useful - particularly for catchments models with multiple functional units: value recycling.
If you provide a list with few values than are required, the system will start again from the start of the list.
So, for example, the following code will assign the three values: [0.5,0.75,1.0]
In [28]:
v.model.link.routing.set_param_values('RoutingPower',[0.5,0.75,1.0],fromList=True)
Out[28]:
Check the Feature Table to see the effect.
Note: You can run these scripts with the Feature Table open and the model will be updated - but the feature table won't reflect the new values until you Cancel the feature table and reopen it.
In [ ]:
As mentioned, everything under v.model works by sending an IronPython script to Source to be run within the Source software itself.
IronPython is a native, .NET, version of Python and hence can access all the classes and objects that make up Source.
When you call a function witnin v.model, veneer-py is generating an IronPython script for Source.
To this point, we haven't seen what these IronPython scripts look like - they are hidden from view. We can see the scripts that get sent to Source by setting the option veneer.general.PRINT_SCRIPTS=True
In [47]:
veneer.general.PRINT_SCRIPTS=True
v.model.link.routing.get_models(links=['Default Link #3','Default Link #4'])
Out[47]:
In [48]:
veneer.general.PRINT_SCRIPTS=False
Writing these IronPython scripts from scratch requires an understanding of the internal data structures of Source. The functions under v.model are designed to shield you from these details.
That said, if you have an idea of the data structures, you may wish to try writing IronPython scripts, OR, try working with some of the lower-level functionality offered in v.model.
Most of the v.model functions that we've used, are ultimately based upon two, low level thems:
v.model.get andv.model.setBoth get and set expect a query to perform on a Source scenario object. Structuring this query is where an understanding of Source data structures comes in.
For example, the following query will return the number of nodes in the network. (We'll use the PRINT_SCRIPTS option to show how the query translates to a script):
In [49]:
veneer.general.PRINT_SCRIPTS=True
num_nodes = v.model.get('scenario.Network.Nodes.Count()')
num_nodes
Out[49]:
The follow example returns the names of each node in the network. The .* notation tells veneer-py to generate a loop over every element in a collection
In [56]:
node_names = v.model.get('scenario.Network.Nodes.*Name')
node_names
Out[56]:
You can see from the script output that veneer-py has generated a Python for loop to iterate over the nodes:
for i_0 in scenario.Network.Nodes:
There are other characteristics in there, such as ignoring exceptions - this is a common default used in v.model to silently skip nodes/links/catchments/etc that don't have a particular property.
The same query approach can work for set, which can set a particular property (on one or more objects) to a particular value (which can be the same value everywhere, or drawn from a list)
In [60]:
# Generate a new name for each node (based on num_nodes)
names = ['New Name %d'%i for i in range(num_nodes)]
names
Out[60]:
In [61]:
v.model.set('scenario.Network.Nodes.*Name',names,fromList=True,literal=True)
Out[61]:
If you look at the Source model now (you may need to trigger a redraw by resizing the window), all the nodes have been renamed.
(Lets reset the names - note how we saved node_names earlier on!)
In [62]:
v.model.set('scenario.Network.Nodes.*Name',node_names,fromList=True,literal=True)
Out[62]:
Note: The literal=True option is currently necessary setting text properties using v.model.set. This tells the IronPython generator to wrap the strings in quotes in the final script. Otherwise, IronPython would be looking for symbols (eg classes) with the same names
The examples of v.model.get and v.model.set illustrate some of the low level functionality for manipulating the source model.
The earlier, high-level, functions (eg v.model.link.routing.set_param_values) take care of computing the query string for you, including context dependent code such as searching for links of a particular name, or nodes of a particular type. They then call the lower level functions, which takes care of generating the actual IronPython script.
The v.model namespace is gradually expanding with new capabilities and functions - but at their essence, most new functions provide a high level wrapper, around v.model.get and v.model.set for some new area of the Source data structures. So, for example, you could envisage a v.model.resource_assessment which provides high level wrappers around resource assessment functionality.
Writing the high level wrappers (as with writing the query strings for v.model.get/set) requires an understanding of the internal data structures of Source. You can get this from the C# code for Source, or, to a degree, from a help function v.model.sourceHelp.
Lets say you want to discover how to change the description of the scenario (say, to automatically add a note about the changes made by your script)
Start, by asking for help on 'scenario' and explore from there
In [5]:
veneer.general.PRINT_SCRIPTS=False
v.model.sourceHelp('scenario')
Out[5]:
This tells you everything that is available on a Source scenario. It's a lot, but Description looks promising:
In [4]:
existing_description = v.model.get('scenario.Description')
existing_description
OK. It looks like there is no description in the existing scenario. Lets set one
In [5]:
v.model.set('scenario.Description','Model modified by script',literal=True)
Out[5]:
In [ ]:
Lets look at a simple model building example.
We will test out different routing parameters, by setting up a scenario with several parallel networks. Each network will consist of an Inflow Node and a Gauge Node, joined by a Storage Routing link.
The inflows will all use the same time series of flows, so the only difference will be the routing parameters.
To proceed,
Now, create a new veneer client (creatively called v2 here)
In [2]:
v2 = veneer.Veneer(port=9877)
And check that the network has nothing in it at the moment
In [3]:
v2.network()
Out[3]:
We can create nodes with v.model.node.create
In [4]:
help(v2.model.node.create)
There are also functions to create different node types:
In [5]:
help(v2.model.node.new_gauge)
First, we'll do a bit of a test run. Ultimately, we'll want to create a number of such networks - and the nodes will definitely need unique names then
In [ ]:
In [27]:
In [6]:
loc = [10,10]
v2.model.node.new_inflow('The Inflow',schematic_location=loc,location=loc)
Out[6]:
In [7]:
loc = [20,10]
v2.model.node.new_gauge('The Gauge',schematic_location=loc,location=loc)
Out[7]:
Note: At this stage (and after some frustration) we can't set the location of the node on the schematic. We can set the 'geographic' location - which doesn't have to be true geographic coordinates, so that's what we'll do here.
Creating a link can be done with v2.model.link.create
In [8]:
help(v2.model.link.create)
In [9]:
v2.model.link.create('The Inflow','The Gauge','The Link')
Out[9]:
Now, lets look at the information from v2.network() to see that it's all there. (We should also see the model in the geographic view)
In [10]:
v2.network().as_dataframe()
Out[10]:
Now, after all that, we'll delete everything we've created and then recreate it all in a loop to give us parallel networks
In [11]:
v2.model.node.remove('The Inflow')
v2.model.node.remove('The Gauge')
Out[11]:
So, now we can create (and delete) nodes and links, lets create multiple parallel networks, to test out our flow routing parameters. We'll create 20, because we can!
In [ ]:
In [12]:
num_networks=20
In [13]:
for i in range(1,num_networks+1): # Loop from 1 to 20
veneer.log('Creating network %d'%i)
x = i
loc_inflow = [i,10]
loc_gauge = [i,0]
name_inflow = 'Inflow %d'%i
name_gauge = 'Gauge %d'%i
v2.model.node.new_inflow(name_inflow,location=loc_inflow,schematic_location=loc_inflow)
v2.model.node.new_gauge(name_gauge,location=loc_gauge,schematic_location=loc_gauge)
# Create the link
name_link = 'Link %d'%i
v2.model.link.create(name_inflow,name_gauge,name_link)
# Set the routing type to storage routing (we *could* do this at the end, outside the loop)
v2.model.link.routing.set_models('RiverSystem.Flow.StorageRouting',links=name_link)
We'll use one of the flow files from the earlier model to drive each of our inflow nodes. We need to know where that data is. Here, I'm assuming its in the ExampleProject directory within the same directory as this notebook. We'll need the absolute path for Source, and the Python os package helps with this type of filesystem operation
In [14]:
import os
In [15]:
os.path.exists('ExampleProject/Fish_G_flow.csv')
Out[15]:
In [16]:
absolute_path = os.path.abspath('ExampleProject/Fish_G_flow.csv')
absolute_path
Out[16]:
In [ ]:
We can use v.model.node.assign_time_series to attach a time series of inflows to the inflow node. We could have done this in the for loop, one node at a time, but, like set_param_values we can assign time series to multiple nodes at once.
One thing that we do need to know is the parameter that we're assigning the time series to (because, after all, this could be any type of node - veneer-py doesn't know at this stage). We can find the model type, then check v.model.find_parameters and, if that doesn't work, v.model.find_inputs:
In [19]:
v2.model.node.get_models(nodes='Inflow 1')
Out[19]:
In [20]:
v2.model.find_parameters('RiverSystem.Nodes.Inflow.InjectedFlow')
Out[20]:
In [21]:
v2.model.find_inputs('RiverSystem.Nodes.Inflow.InjectedFlow')
Out[21]:
So 'Flow' it is!
In [22]:
v2.model.node.assign_time_series('Flow',absolute_path,'Inflows')
Out[22]:
Almost there.
Now, lets set a range of storage routing parameters (much like we did before)
In [23]:
power_vals = np.arange(1.0,0.0,-1.0/num_networks)
power_vals
Out[23]:
And assign those to the links
In [24]:
v2.model.link.routing.set_param_values('RoutingConstant',86400.0)
v2.model.link.routing.set_param_values('RoutingPower',power_vals,fromList=True)
Out[24]:
Now, configure recording
In [25]:
v2.configure_recording(disable=[{}],enable=[{'RecordingVariable':'Downstream Flow Volume'}])
And one last thing - work out the time period for the run from the inflow time series
In [26]:
inflow_ts = pd.read_csv(absolute_path,index_col=0)
start,end=inflow_ts.index[[0,-1]]
start,end
Out[26]:
That looks a bit much. Lets run for a year
In [27]:
v2.run_model(start='01/01/1999',end='31/12/1999')
Out[27]:
Now, we can retrieve some results. Because we used a naming convention for all the nodes, its possible to grab relevant results using those conventions
In [28]:
upstream = v2.retrieve_multiple_time_series(criteria={'RecordingVariable':'Downstream Flow Volume','NetworkElement':'Inflow.*'})
downstream = v2.retrieve_multiple_time_series(criteria={'RecordingVariable':'Downstream Flow Volume','NetworkElement':'Gauge.*'})
In [29]:
downstream[['Gauge 1:Downstream Flow Volume','Gauge 20:Downstream Flow Volume']].plot(figsize=(10,10))
Out[29]:
If you'd like to change and rerun this example, the following code block can be used to delete all the existing nodes. (Or, just start a new project in Source)
In [30]:
#nodes = v2.network()['features'].find_by_feature_type('node')._all_values('name')
#for n in nodes:
# v2.model.node.remove(n)
This session has looked at structural modifications of Source using Veneer, veneer-py and the use of IronPython scripts that run within Source.
Writing IronPython scripts requires a knowledge of internal Source data structures, but there is a growing collection of helper functions, under the v.model namespace to assist.
In [ ]:
In [ ]:
In [ ]: