Retrieving results from functional units

Retrieving results for functional units has been problematic because Functional unit results don't fit neatly within the three main criteria (NetworkElement - which maps to catchment, RecordingElement and RecordingVariable).

Long story short - the typical behaviour is that you get N subcatchments M functional units M functional units.

So, in the example here, 10 catchments, 5 functional units leads to 10 x 5 x 5 = 250 time series being retrieved, rather than the expected 50.

This can partly be dealt with by using an appropriate column naming function (veneer.name_for_fu_and_sc) and then filtering the results, but its still annoying, slow and wastes memory.

In recent versions of Veneer, you can now request results for a specific Functional Unit. You need to call v.retrieve_multiple_time_series multiple times, but you end up with just what you're after


In [54]:
import veneer

In [55]:
v = veneer.Veneer()

In [56]:
set(v.model.catchment.get_functional_unit_types())


Out[56]:
{'GeneratedFunctionalUnit0',
 'GeneratedFunctionalUnit1',
 'GeneratedFunctionalUnit2',
 'GeneratedFunctionalUnit3',
 'GeneratedFunctionalUnit4'}

In [57]:
v.drop_all_runs()

In [58]:
v.configure_recording(enable=[{'RecordingVariable':'Total Flow'}])

In [59]:
v.run_model()


Out[59]:
(302, 'runs/2')

Standard behaviour

Functional units results repeated.


In [60]:
res = v.retrieve_multiple_time_series(criteria={'RecordingVariable':'Total Flow'})
res[:10]


Out[60]:
Catchment for node #0:Total Flow Catchment for node #0:Total Flow 1 Catchment for node #0:Total Flow 2 Catchment for node #0:Total Flow 3 Catchment for node #0:Total Flow 4 Catchment for node #1:Total Flow Catchment for node #1:Total Flow 1 Catchment for node #1:Total Flow 2 Catchment for node #1:Total Flow 3 Catchment for node #1:Total Flow 4 ... Catchment for node #8:Total Flow Catchment for node #8:Total Flow 1 Catchment for node #8:Total Flow 2 Catchment for node #8:Total Flow 3 Catchment for node #8:Total Flow 4 Catchment for node #9:Total Flow Catchment for node #9:Total Flow 1 Catchment for node #9:Total Flow 2 Catchment for node #9:Total Flow 3 Catchment for node #9:Total Flow 4
1900-01-01 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-02 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-03 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-04 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-05 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-06 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-07 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-08 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-09 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

10 rows × 50 columns


In [61]:
len(res.columns)


Out[61]:
50

In [62]:
res.columns


Out[62]:
Index(['Catchment for node #0:Total Flow',
       'Catchment for node #0:Total Flow 1',
       'Catchment for node #0:Total Flow 2',
       'Catchment for node #0:Total Flow 3',
       'Catchment for node #0:Total Flow 4',
       'Catchment for node #1:Total Flow',
       'Catchment for node #1:Total Flow 1',
       'Catchment for node #1:Total Flow 2',
       'Catchment for node #1:Total Flow 3',
       'Catchment for node #1:Total Flow 4',
       'Catchment for node #2:Total Flow',
       'Catchment for node #2:Total Flow 1',
       'Catchment for node #2:Total Flow 2',
       'Catchment for node #2:Total Flow 3',
       'Catchment for node #2:Total Flow 4',
       'Catchment for node #3:Total Flow',
       'Catchment for node #3:Total Flow 1',
       'Catchment for node #3:Total Flow 2',
       'Catchment for node #3:Total Flow 3',
       'Catchment for node #3:Total Flow 4',
       'Catchment for node #4:Total Flow',
       'Catchment for node #4:Total Flow 1',
       'Catchment for node #4:Total Flow 2',
       'Catchment for node #4:Total Flow 3',
       'Catchment for node #4:Total Flow 4',
       'Catchment for node #5:Total Flow',
       'Catchment for node #5:Total Flow 1',
       'Catchment for node #5:Total Flow 2',
       'Catchment for node #5:Total Flow 3',
       'Catchment for node #5:Total Flow 4',
       'Catchment for node #6:Total Flow',
       'Catchment for node #6:Total Flow 1',
       'Catchment for node #6:Total Flow 2',
       'Catchment for node #6:Total Flow 3',
       'Catchment for node #6:Total Flow 4',
       'Catchment for node #7:Total Flow',
       'Catchment for node #7:Total Flow 1',
       'Catchment for node #7:Total Flow 2',
       'Catchment for node #7:Total Flow 3',
       'Catchment for node #7:Total Flow 4',
       'Catchment for node #8:Total Flow',
       'Catchment for node #8:Total Flow 1',
       'Catchment for node #8:Total Flow 2',
       'Catchment for node #8:Total Flow 3',
       'Catchment for node #8:Total Flow 4',
       'Catchment for node #9:Total Flow',
       'Catchment for node #9:Total Flow 1',
       'Catchment for node #9:Total Flow 2',
       'Catchment for node #9:Total Flow 3',
       'Catchment for node #9:Total Flow 4'],
      dtype='object')

Slight improvement

Add functional unit to the column name. Same amount of data, but at least you can distinguish the data


In [63]:
res = v.retrieve_multiple_time_series(criteria={'RecordingVariable':'Total Flow'},name_fn=veneer.name_for_fu_and_sc)
res[:10]


Out[63]:
GeneratedFunctionalUnit0: Catchment for node #0 GeneratedFunctionalUnit0: Catchment for node #1 GeneratedFunctionalUnit0: Catchment for node #2 GeneratedFunctionalUnit0: Catchment for node #3 GeneratedFunctionalUnit0: Catchment for node #4 GeneratedFunctionalUnit0: Catchment for node #5 GeneratedFunctionalUnit0: Catchment for node #6 GeneratedFunctionalUnit0: Catchment for node #7 GeneratedFunctionalUnit0: Catchment for node #8 GeneratedFunctionalUnit0: Catchment for node #9 ... GeneratedFunctionalUnit4: Catchment for node #0 GeneratedFunctionalUnit4: Catchment for node #1 GeneratedFunctionalUnit4: Catchment for node #2 GeneratedFunctionalUnit4: Catchment for node #3 GeneratedFunctionalUnit4: Catchment for node #4 GeneratedFunctionalUnit4: Catchment for node #5 GeneratedFunctionalUnit4: Catchment for node #6 GeneratedFunctionalUnit4: Catchment for node #7 GeneratedFunctionalUnit4: Catchment for node #8 GeneratedFunctionalUnit4: Catchment for node #9
1900-01-01 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-02 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-03 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-04 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-05 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-06 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-07 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-08 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-09 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

10 rows × 50 columns


In [64]:
res.columns


Out[64]:
Index(['GeneratedFunctionalUnit0: Catchment for node #0',
       'GeneratedFunctionalUnit0: Catchment for node #1',
       'GeneratedFunctionalUnit0: Catchment for node #2',
       'GeneratedFunctionalUnit0: Catchment for node #3',
       'GeneratedFunctionalUnit0: Catchment for node #4',
       'GeneratedFunctionalUnit0: Catchment for node #5',
       'GeneratedFunctionalUnit0: Catchment for node #6',
       'GeneratedFunctionalUnit0: Catchment for node #7',
       'GeneratedFunctionalUnit0: Catchment for node #8',
       'GeneratedFunctionalUnit0: Catchment for node #9',
       'GeneratedFunctionalUnit1: Catchment for node #0',
       'GeneratedFunctionalUnit1: Catchment for node #1',
       'GeneratedFunctionalUnit1: Catchment for node #2',
       'GeneratedFunctionalUnit1: Catchment for node #3',
       'GeneratedFunctionalUnit1: Catchment for node #4',
       'GeneratedFunctionalUnit1: Catchment for node #5',
       'GeneratedFunctionalUnit1: Catchment for node #6',
       'GeneratedFunctionalUnit1: Catchment for node #7',
       'GeneratedFunctionalUnit1: Catchment for node #8',
       'GeneratedFunctionalUnit1: Catchment for node #9',
       'GeneratedFunctionalUnit2: Catchment for node #0',
       'GeneratedFunctionalUnit2: Catchment for node #1',
       'GeneratedFunctionalUnit2: Catchment for node #2',
       'GeneratedFunctionalUnit2: Catchment for node #3',
       'GeneratedFunctionalUnit2: Catchment for node #4',
       'GeneratedFunctionalUnit2: Catchment for node #5',
       'GeneratedFunctionalUnit2: Catchment for node #6',
       'GeneratedFunctionalUnit2: Catchment for node #7',
       'GeneratedFunctionalUnit2: Catchment for node #8',
       'GeneratedFunctionalUnit2: Catchment for node #9',
       'GeneratedFunctionalUnit3: Catchment for node #0',
       'GeneratedFunctionalUnit3: Catchment for node #1',
       'GeneratedFunctionalUnit3: Catchment for node #2',
       'GeneratedFunctionalUnit3: Catchment for node #3',
       'GeneratedFunctionalUnit3: Catchment for node #4',
       'GeneratedFunctionalUnit3: Catchment for node #5',
       'GeneratedFunctionalUnit3: Catchment for node #6',
       'GeneratedFunctionalUnit3: Catchment for node #7',
       'GeneratedFunctionalUnit3: Catchment for node #8',
       'GeneratedFunctionalUnit3: Catchment for node #9',
       'GeneratedFunctionalUnit4: Catchment for node #0',
       'GeneratedFunctionalUnit4: Catchment for node #1',
       'GeneratedFunctionalUnit4: Catchment for node #2',
       'GeneratedFunctionalUnit4: Catchment for node #3',
       'GeneratedFunctionalUnit4: Catchment for node #4',
       'GeneratedFunctionalUnit4: Catchment for node #5',
       'GeneratedFunctionalUnit4: Catchment for node #6',
       'GeneratedFunctionalUnit4: Catchment for node #7',
       'GeneratedFunctionalUnit4: Catchment for node #8',
       'GeneratedFunctionalUnit4: Catchment for node #9'],
      dtype='object')

Better - use FunctionalUnit criteria

Recent versions of Veneer support the FunctionalUnit criteria. Note this is only available on retrieve_multiple_time_series - not available on configure_recording.

This version is bundled with Source 4.6 onwards and available in custom Veneer builds for earlier versions of Source


In [48]:
v.retrieve_multiple_time_series?


Signature: v.retrieve_multiple_time_series(run='latest', run_data=None, criteria={}, timestep='daily', name_fn=<function name_element_variable at 0x00000185B44641E0>)
Docstring:
Retrieve multiple time series from a run according to some criteria.

Return all time series in a single Pandas DataFrame with date time index.

you can an index of run results via run_data. If you don't the method will first retrieve an
index based on the value of the run parameter (default='latest')

criteria should be regexps for the fields in a Veneer time series record:
  * NetworkElement
  * RecordingElement
  * RecordingVariable
  * TimeSeriesName
  * TimeSeriesUrl
  * FunctionalUnit

These criteria are used to identify which time series to retrieve.

timestep should be one of 'daily' (default), 'monthly', 'annual'.
*WARNING*: The monthly and annual option uses the corresponding option in the Veneer plugin, which ALWAYS SUMS values,
regardless of units. So, if you retrieve a rate variable (eg m^3/s) those values will be summed and you will need to
correct this manually in the returned DataFrame.

All retrieved time series are returned in a single Data Frame.

You can specify a function for naming the columns of the Data Frame using name_fn. This function should take
the results summary (from the index) and return a string. Example functions include:
  * veneer.name_time_series (uses the full name of the time series, as provided by Source)
  * veneer.name_element_variable (DEFAULT: users the name of the network element and the name of the variable)
  * veneer.name_for_location (just use the name of the network element)
  * veneer.name_for_variable (just use the name of the variable)
File:      d:\src\projects\py\veneer-py\veneer\general.py
Type:      method

In [52]:
res = v.retrieve_multiple_time_series(criteria={'RecordingVariable':'Total Flow','FunctionalUnit':'GeneratedFunctionalUnit0'},
                                      name_fn=veneer.name_for_fu_and_sc)
res[:10]


Out[52]:
GeneratedFunctionalUnit0: Catchment for node #0 GeneratedFunctionalUnit0: Catchment for node #1 GeneratedFunctionalUnit0: Catchment for node #2 GeneratedFunctionalUnit0: Catchment for node #3 GeneratedFunctionalUnit0: Catchment for node #4 GeneratedFunctionalUnit0: Catchment for node #5 GeneratedFunctionalUnit0: Catchment for node #6 GeneratedFunctionalUnit0: Catchment for node #7 GeneratedFunctionalUnit0: Catchment for node #8 GeneratedFunctionalUnit0: Catchment for node #9
1900-01-01 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-02 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-03 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-04 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-05 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-06 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-07 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-08 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-09 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1900-01-10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

In [53]:
res.columns


Out[53]:
Index(['GeneratedFunctionalUnit0: Catchment for node #0',
       'GeneratedFunctionalUnit0: Catchment for node #1',
       'GeneratedFunctionalUnit0: Catchment for node #2',
       'GeneratedFunctionalUnit0: Catchment for node #3',
       'GeneratedFunctionalUnit0: Catchment for node #4',
       'GeneratedFunctionalUnit0: Catchment for node #5',
       'GeneratedFunctionalUnit0: Catchment for node #6',
       'GeneratedFunctionalUnit0: Catchment for node #7',
       'GeneratedFunctionalUnit0: Catchment for node #8',
       'GeneratedFunctionalUnit0: Catchment for node #9'],
      dtype='object')

In [ ]: