All NILM datasets consists of various groupings of electricity meters. We can group the meters by house. Or by the type of appliance they are directly connected to. Or by sample rate. Or by whether the meter is a whole-house "site meter" or an appliance-level submeter, or a circuit-level submeter.
In NILMTK, one of the key classes is MeterGroup which stores a list of meters and allows us to select a subset of meters, aggregate power from all meters and many other functions.
When we first open a DataSet, NILMTK creates several MeterGroup objects. There's nilmtk.global_meter_group which holds every meter currently loaded (including from multiple datasets if you have opened more than one dataset). There is also one MeterGroup per building (which live in the Building.elec attribute). We also nested MeterGroups for aggregating together split-phase mains, 3-phase mains and dual-supply (240 volt) appliances in North American and Canadian datasets. For example, here is the MeterGroup for building 1 in REDD:
NOTE: If you are on Windows, remember to escape the back-slashes, use forward-slashs, or use raw-strings when passing paths in Python, e.g. one of the following would work:
redd = DataSet('c:\\data\\redd.h5')
redd = DataSet('c:/data/redd.h5')
redd = DataSet(r'c:\data\redd.h5')
In [1]:
%matplotlib inline
In [2]:
from matplotlib import rcParams
import matplotlib.pyplot as plt
import pandas as pd
import nilmtk
from nilmtk import DataSet, MeterGroup
plt.style.use('ggplot')
rcParams['figure.figsize'] = (13, 10)
redd = DataSet('/data/redd.h5')
elec = redd.buildings[1].elec
elec
Out[2]:
Note that there are two nested MeterGroups: one for the electric oven, and one for the washer dryer (both of which are 240 volt appliances and have two meters per appliance):
In [3]:
elec.nested_metergroups()
Out[3]:
Putting these meters into a MeterGroup allows us to easily sum together the power demand recorded by both meters to get the total power demand for the entire appliance (but it's also very easy to see the individual meter power demand too).
We can easily get a MeterGroup of either the submeters or the mains:
In [4]:
elec.mains()
Out[4]:
We can easily get the power data for both mains meters summed together:
In [5]:
elec.mains().power_series_all_data().head()
Out[5]:
In [6]:
elec.submeters()
Out[6]:
Let's work out the proportion of energy submetered in REDD building 1:
In [7]:
elec.proportion_of_energy_submetered()
Out[7]:
Note that NILMTK has raised a warning that Mains uses a different type of power measurement than all the submeters, so it's not an entirely accurate comparison. Which raises the question: which type of power measurements are used for the mains and submeters? Let's find out...
In [8]:
mains = elec.mains()
In [9]:
mains.available_ac_types('power')
Out[9]:
In [10]:
elec.submeters().available_ac_types('power')
Out[10]:
In [11]:
next(elec.load())
Out[11]:
In [12]:
elec.mains().total_energy() # returns kWh
Out[12]:
In [13]:
energy_per_meter = elec.submeters().energy_per_meter() # kWh, again
energy_per_meter
Out[13]:
column headings are the ElecMeter instance numbers.
The function fraction_per_meter does the same thing as energy_per_submeter but returns the fraction of energy per meter.
Let's make a new MeterGroup which only contains the ElecMeters which used more than 20 kWh:
In [14]:
# energy_per_meter is a DataFrame where each row is a
# power type ('active', 'reactive' or 'apparent').
# All appliance meters in REDD are record 'active' so just select
# the 'active' row:
energy_per_meter = energy_per_meter.loc['active']
more_than_20 = energy_per_meter[energy_per_meter > 20]
more_than_20
Out[14]:
In [15]:
instances = more_than_20.index
instances
Out[15]:
In [16]:
fraction = elec.submeters().fraction_per_meter().dropna()
In [17]:
# Create convenient labels
labels = elec.get_labels(fraction.index)
plt.figure(figsize=(10,30))
fraction.plot(kind='pie', labels=labels);
We can get the wiring diagram for the MeterGroup:
In [18]:
elec.draw_wiring_graph()
Out[18]:
It's not very pretty but it shows that meters (1,2) (the site meters) are upstream of all other meters.
Buildings in REDD have only two levels in their meter hierarchy (mains and submeters). If there were more than two levels then it might be useful to get only the meters immediately downstream of mains:
In [19]:
elec.meters_directly_downstream_of_mains()
Out[19]:
In [20]:
#sns.set_palette("Set3", n_colors=12)
# Set a threshold to remove residual power noise when devices are off
elec.plot_when_on(on_power_threshold = 40)
Out[20]:
The ElecMeter class represents a single electricity meter. Each ElecMeter has a list of associated Appliance objects. ElecMeter has many of the same stats methods as MeterGroup such as total_energy and available_power_ac_types and power_series and power_series_all_data. We will now explore some more stats functions (many of which are also available on MeterGroup)...
In [21]:
fridge_meter = elec['fridge']
In [22]:
fridge_meter.upstream_meter() # happens to be the mains meter group!
Out[22]:
In [23]:
fridge_meter.device
Out[23]:
If the metadata specifies that a meter has multiple meters connected to it then one of those can be specified as the 'dominant' appliance, and this appliance can be retrieved with this method:
In [24]:
fridge_meter.dominant_appliance()
Out[24]:
In [25]:
fridge_meter.total_energy() # kWh
Out[25]:
If we plot the raw power data then we see there is one large gap where, supposedly, the metering system was not working. (if we were to zoom in then we'd see lots of smaller gaps too):
In [26]:
fridge_meter.plot()
Out[26]:
We can automatically identify the 'good sections' (i.e. the sections where every pair of consecutive samples is less than max_sample_period specified in the dataset metadata):
In [27]:
good_sections = fridge_meter.good_sections(full_results=True)
# specifying full_results=False would give us a simple list of
# TimeFrames. But we want the full GoodSectionsResults object so we can
# plot the good sections...
In [28]:
good_sections.plot()
Out[28]:
The blue chunks show where the data is good. The white gap is the large gap seen in the raw power data. There are lots of smaller gaps that we cannot see at this zoom level.
We can also see the exact sections identified:
In [29]:
good_sections.combined()
Out[29]:
As well as large gaps appearing because the entire system is down, we also get frequent small gaps from wireless sensors dropping data. This is sometimes called 'dropout'. The dropout rate is a number between 0 and 1 which specifies the proportion of missing samples. A dropout rate of 0 means no samples are missing. A value of 1 would mean all samples are missing:
In [30]:
fridge_meter.dropout_rate()
Out[30]:
Note that the dropout rate has gone down (which is good!) now that we are ignoring the gaps. This value is probably more representative of the performance of the wireless system.
We use ElecMeter.select_using_appliances() to select a new MeterGroup using an metadata field. For example, to get all the washer dryers in the whole of the REDD dataset:
In [31]:
import nilmtk
nilmtk.global_meter_group.select_using_appliances(type='washer dryer')
Out[31]:
Or select multiple appliance types:
In [32]:
elec.select_using_appliances(type=['fridge', 'microwave'])
Out[32]:
Or all appliances in the 'heating' category:
In [33]:
nilmtk.global_meter_group.select_using_appliances(category='heating')
Out[33]:
Or all appliances in building 1 with a single-phase induction motor(!):
In [34]:
nilmtk.global_meter_group.select_using_appliances(building=1, category='single-phase induction motor')
Out[34]:
(NILMTK imports the 'common metadata' from the NILM Metadata project, which includes a wide range of different category taxonomies)
In [35]:
nilmtk.global_meter_group.select_using_appliances(building=2, category='laundry appliances')
Out[35]:
In [36]:
elec.select(device_model='REDD_whole_house')
Out[36]:
In [37]:
elec.select(sample_period=3)
Out[37]:
We use [] to retrive a single ElecMeter from a MeterGroup.
In [38]:
elec['fridge']
Out[38]:
Appliances are uniquely identified within a building by a type (fridge, kettle, television, etc.) and an instance number. If we do not specify an instance number then ElecMeter retrieves instance 1 (instance numbering starts from 1). If you want a different instance then just do this:
In [39]:
elec.select_using_appliances(type='fridge')
Out[39]:
In [40]:
elec['light', 2]
Out[40]:
To uniquely identify an appliance in nilmtk.global_meter_group then we must specify the dataset name, building instance number, appliance type and appliance instance in a dict:
In [41]:
import nilmtk
nilmtk.global_meter_group[{'dataset': 'REDD', 'building': 1, 'type': 'fridge', 'instance': 1}]
Out[41]:
get ElecMeter with instance = 1:
In [42]:
elec[1]
Out[42]:
ElecMeter and Appliance instance numbers uniquely identify the meter or appliance type within the building, not globally. To uniquely identify a meter globally, we need three keys:
In [43]:
from nilmtk.elecmeter import ElecMeterID
# ElecMeterID is a namedtuple for uniquely identifying each ElecMeter
nilmtk.global_meter_group[ElecMeterID(instance=8, building=1, dataset='REDD')]
Out[43]:
We can also select a single, existing nested MeterGroup. There are two ways to specify a nested MeterGroup:
In [44]:
elec[[ElecMeterID(instance=3, building=1, dataset='REDD'),
ElecMeterID(instance=4, building=1, dataset='REDD')]]
Out[44]:
In [45]:
elec[ElecMeterID(instance=(3,4), building=1, dataset='REDD')]
Out[45]:
We can also specify the mains by asking for meter instance 0:
In [46]:
elec[ElecMeterID(instance=0, building=1, dataset='REDD')]
Out[46]:
which is equivalent to elec.mains():
In [47]:
elec.mains() == elec[ElecMeterID(instance=0, building=1, dataset='REDD')]
Out[47]:
In [48]:
redd.set_window(start='2011-04-21', end='2011-04-22')
elec.plot();
plt.xlabel("Time");
In [49]:
from pandas.plotting import autocorrelation_plot
In [50]:
elec.mains().plot_autocorrelation();
In [51]:
fridges_restricted = nilmtk.global_meter_group.select_using_appliances(type='fridge')
daily_energy = pd.Series([meter.average_energy_per_period(offset_alias='D')
for meter in fridges_restricted.meters])
# daily_energy.plot(kind='hist');
# plt.title('Histogram of daily fridge energy');
# plt.xlabel('energy (kWh)');
# plt.ylabel('occurences');
# plt.legend().set_visible(False)
daily_energy
Out[51]:
In [52]:
correlation_df = elec.pairwise_correlation()
correlation_df
Out[52]: