Basically, the first main input data is the raw data sensors. These data can be acquired using a data acquisition device (DAQ) through analog channels (e.g. weigh sensors, temperature sensors, etc) and/or digital channels (e.g., inductive loops).
The three more wideley piezo-eletric weigh sensors used are piezo-ceramic, piezo-polymer and piezo-electric (Jiang, 2009).
The storing the raw sensor data allows studying of the input signals and validating weigh methods. In COST 323 (Jacob et al., 2009), it was not found any description about the raw data layout file. By the way, this data can be represented by a matrix using as a first column a index with time instant, it can be represented by microseconds in floating point format and it is followed by other columns representing each sensor data.
On one file it can be saved any measurements of vehicle's run, e.g. the researcher can create one file per day and on each file all vehicle's run, with respecting to the date of the file. Each vehicle's run should be saved on a specific dataset. The main idea of these standards is promoting a best practice to store and share weigh-in-motion data.
The filename should have be informative, respecting the date, site and lane and the organization type of the dataset. If the file contains measurements from more than one site so the site identification number should be 000. The same idea should be used to lane identification number. The date field from the filename should contain the initial date time of the period. If it is necessary, inform the initial time too (optional). The standard structure proposed is:
wim_t_sss_ll_yyyymmdd[_hhMMSS]E.g. wim_day_001_01_20174904_004936. When:
For each vehicle's run, it should be created a new dataset. The dataset name should contain site identification number, lane identification number, date and time. The standard structure proposed is:
run_sss_ll_yyyymmdd_hhMMSSE.g. run_001_01_20174904_004936. When run is a fixed text. The other parts in dataset name can be explained as in file name standard.
Each dataset contains information from signal data. The dataset should contain some extra information to allow data post-processing. The columns on the dataset should be index and data from analog channels and digital channels. The standard for column names should be:
{t}{n}Where {t} means the channel type (i.e. can be set as a for analog, or d for digital) and {n} means the number of the channel (e.g. a1).
The main extra information that should be saved on the dataset is:
The algorithms presented here was written in Python language. If it is necessary to use another language would be easy to convert or rewrite this code in any language.
Storage Data module should be able to write and read data from hdf5 file with a simple approach, in other words, it should be easy for anybody to manipulate and understand this data using other languages.
In [1]:
from IPython.display import display
from datetime import datetime
from matplotlib import pyplot as plt
from scipy import misc
import h5py
import json
import numpy as np
import os
import pandas as pd
import sys
In [2]:
# local
sys.path.insert(0, os.path.dirname(os.getcwd()))
from pywim.utils.dsp.synthetic_data.sensor_data import gen_truck_raw_data
In [3]:
# generates a synthetic data
sample_rate = 2000
sensors_distance = [1, 2]
data = gen_truck_raw_data(
sample_rate=sample_rate, speed=20, vehicle_layout='O--O------O-',
sensors_distance=sensors_distance, p_signal_noise=100.0
)
data.plot()
plt.show()
In [4]:
data.head()
Out[4]:
In [5]:
date_time = datetime.now()
site_id = '001'
lane_id = '01'
collection_type = 'day' # stored per day
f_id = 'wim_{}_{}_{}_{}'.format(
collection_type, site_id, lane_id,
date_time.strftime('%Y%m%d')
)
f = h5py.File('/tmp/{}.h5'.format(f_id), 'w')
print(f_id)
In [6]:
dset_id = 'run_{}_{}_{}'.format(
site_id, lane_id, date_time.strftime('%Y%M%d_%H%M%S')
)
print(dset_id)
In [7]:
dset = f.create_dataset(
dset_id, shape=(data.shape[0],),
dtype=np.dtype([
(k, float) for k in ['index'] + list(data.keys())
])
)
dset['index'] = data.index
for k in data.keys():
dset[k] = data[k]
In [8]:
# check if all values are the same
df = pd.DataFrame(dset[tuple(data.keys())], index=dset['index'])
np.all(df == data)
Out[8]:
In [9]:
dset.attrs['sample_rate'] = sample_rate
dset.attrs['date_time'] = date_time.strftime('%Y-%M-%d %H:%M:%S')
dset.attrs['site_id'] = site_id
dset.attrs['lane_id'] = lane_id
dset.attrs['temperature'] = 28.5
dset.attrs['license_plate'] = 'AAA9999' # license plate number
dset.attrs['calibration_constant'] = [0.98, 0.99, 0.75]
dset.attrs['sensors_distance'] = sensors_distance
dset.attrs['sensor_type'] = 'mixed'
dset.attrs['sensors_layout'] = '|||'
dset.attrs['channel_configuration'] = json.dumps({
'a0': 'polymer', 'a1': 'ceramic', 'a2': 'polymer'
})
# flush its data to disk and close
f.flush()
f.close()
In [10]:
print('/tmp/{}.h5'.format(f_id))
f = h5py.File('/tmp/{}.h5'.format(f_id), 'r')
In [11]:
for dset_id in f.keys():
dset = f[dset_id]
paddle = len(max(dset.attrs, key=lambda v: len(v)))
print('')
print('='*80)
print(dset_id)
print('='*80)
for k in dset.attrs:
print('{}:'.format(k).ljust(paddle, ' '), dset.attrs[k], sep='\t')
pd.DataFrame(dset[dset.dtype.names[1:]], index=dset['index']).plot()
plt.show()
# f.__delitem__(dset_id)
In [12]:
f.flush()
f.close()
Jiang, X. (2009). Improvements in Piezoelectric Sensors and WIM Data Collection Technology (Doctoral dissertation, University of Waterloo, Canada).
Jacob, B., OBrien, E. J., & Jehaes, S. (2002). Weigh-in-Motion of Road Vehicles: Final Report of the COST 323 Action. Laboratoire Central des Ponts et Chaussées Publications, Paris.