Mumodo Demo Notebook -- Update on 24.04.2015
Summary: This notebook describes the basic IO functions to import data from various file types
(c) Dialogue Systems Group, University of Bielefeld
In [1]:
from mumodo.mumodoIO import open_intervalframe_from_increco, open_intervalframe_from_textgrid, \
open_streamframe_from_xiofile, save_intervalframe_to_textgrid, \
save_streamframe_to_xiofile, quantize, open_intervalframe_from_increco
from mumodo.xiofile import XIOFile
from mumodo.increco import IncReco
from mumodo.InstantIO import MFVec2f
import pickle
import pandas as pd
The high level functions allow importing Interval and Stream Frames from files, e.g.
In [2]:
transcriptions = open_intervalframe_from_textgrid("sampledata/test.TextGrid")
The function we just run returns a Python dictionary with the names of the tiers as keys and the IntervalFrames as values. One of these appears to be a PointFrame
In [3]:
transcriptions.keys()
Out[3]:
In [4]:
transcriptions['S']
Out[4]:
In [5]:
transcriptions['CLAPS']
Out[5]:
In order to save back into a textgrid, we use a similar function, packing our Interval and PointFrames in a dict. The line below will save a Praat textgrid with two copies of the 'CLAPS' tier
In [6]:
save_intervalframe_to_textgrid({'the_claps': transcriptions['CLAPS'],
'the_claps_copy': transcriptions['CLAPS']}, 'newtextgrid.TextGrid')
StreamFrames can be loaded from XIO files (see below). It is important to know the sensorname (second argument) as data from many sensors can be stored in a single XIO file. See below (in the XIO files section) how to find out these sensornames.
In [7]:
mystreamframe1 = open_streamframe_from_xiofile('sampledata/test.xio.gz', 'VeniceHubReplay/Venice/Body1')
mystreamframe2 = open_streamframe_from_xiofile('sampledata/test.xio.gz', 'VeniceHubReplay/Kinect/Face')
In [8]:
mystreamframe1[:1]
Out[8]:
In [9]:
mystreamframe2[-1:]
Out[9]:
StreamFrames can be saved back to XIO files as follows. Note that we use exactly the same sensornames, although this is optional.
NOTE: The resulting XIO file will NOT be the same as the input file, due to quantization, see below
In [10]:
#Running the line below creates a new 1.5 MB file and takes about 20 seconds
save_streamframe_to_xiofile({'VeniceHubReplay/Kinect/Face': mystreamframe2,
'VeniceHubReplay/Venice/Body1': mystreamframe1},
'newxiofile.xio.gz')
Instead of saving a StreamFrame to an XIO file or an IntervalFrame to a TextGrid, there are also the following options:
Both of these methods are faster than saving/loading to XIO files, so they are particularly helpful for StreamFrames
When saving/loading from CSV, all objects (except maybe primitives such as floats) are turned into strings , that is, you have to parse the data into objects yourself. This is not so much of a problem for IntervalFrames, which can be safely saved into CSV files.
In [11]:
mystreamframe2.to_csv( "streamframeas.csv" )
streamframefromcsv = pd.DataFrame.from_csv("streamframeas.csv")
assert (mystreamframe2['time'] == streamframefromcsv['time']).all()
In [12]:
#the StreamFrame loaded from disk has type of str for this cell
type(mystreamframe2['FaceEyeLeft'].ix[66199]), type(streamframefromcsv['FaceEyeLeft'].ix[66199])
Out[12]:
In [13]:
#reconstruct the objects in the column
streamframefromcsv['FaceEyeLeft'] = streamframefromcsv['FaceEyeLeft'].map(lambda x: MFVec2f(x))
#check again
type(mystreamframe2['FaceEyeLeft'].ix[66199]), type(streamframefromcsv['FaceEyeLeft'].ix[66199])
Out[13]:
When saving/loading using the pickle module, the objects are preserved
In [14]:
pickle.dump(mystreamframe2, open( "pickledstreamframe", "wb" ) )
unpickledstreamframe = pickle.load( open( "pickledstreamframe", "rb" ) )
assert (mystreamframe2['time'] == unpickledstreamframe['time']).all()
In [15]:
type(mystreamframe2['FaceEyeLeft'].ix[66199]), type(streamframefromcsv['FaceEyeLeft'].ix[66199])
Out[15]:
However, pickle/unpickle needs you to have the object defined identically in the two Python sessions that you save/load. See the pickle module documentation for details
NOTE: The XIO format is inherited into mumodo from the FAME software developed by the AI group, university of Bielefeld and is also used by the venice.hub software, developed by the Dialogue Systems Group. Although the venice.hub format is simpler and newer, mumodo is fully backwards compatible with the original XIO format.
NOTE: Most of the time you will not have to deal with XIO files directly, but it is good to know a little bit about their structure
XIO files are XML files that contain typed, timed events (TTE). The XIOFile class handles such files, e.g. to read an existing XIO file:
In [16]:
f = XIOFile("sampledata/test.xio.gz", 'r')
Let's have a look at a few of the raw lines in the file. Note that 0, 1 are times in milliseconds relative to the start of the file. Because it is very verbose and consumes a lot of disk space, it is typically compressed by means of the gzip module, hence the gz extension. Mumodo can handle both compressed and uncompressed files. This is what the data in the file looks like
In [17]:
for line in f.xio_quicklinegen(0, 1, parsed=False):
print line
Each of these lines represents a typed, timed event (TTE). That is, each event has a timestamp, a type, a value and a sensorname/fieldname combo. The latter part is parsed as follows. The last part of the sensorname attribute (anything after the last "/" becomes the fieldname (and eventually a column in an imported stream frame) while the rest is the sensorname itself. Here are the same lines as above, but now they are parsed. We see that already the strings have been parsed into basic InstantIO objects (see Basic Types demo notebook)
In [18]:
for line in f.xio_quicklinegen(0, 1):
print line
Two typical actions that you may need to do on XIO files are:
The first is very easy, here is an one-liner:
In [19]:
XIOFile("sampledata/test.xio.gz", "r").min_time
Out[19]:
The second one, little bit trickier, requires opening the XIO file in indexed mode, which is a debug mode that pre-parses the entire file. In order to limit this, we only parse a part of the file (1000 lines), if we know that data from all the sensors we want has been logged in this first part:
In [20]:
XIOFile("sampledata/test.xio.gz", indexing=True, maxlines=1000).fieldnames.keys()
Out[20]:
When importing a StreamFrame from an XIOFile, you convert this sequential, asynchronous event stream into a table. It is quite common that data from the same sensor is logged asybchronously, with slightly different timestamps (this is in part due to the way the venice.hub and other loggers work). For example, data from the same sensor could arrive at the following timestamps (which have been made relative to min_time for convenience)
In [21]:
timestamps = []
for line in f.xio_quicklinegen(5000, 5300):
if 'Body' in line['sensorname']:
timestamps.append(line['time'] - f.min_time)
print timestamps
we notice that the sensor outputs 6 values roughly every ~33 ms, which corresponds to 30 frames per second (fps). If we allowed each timestamp to become a row in a StreamFrame, we would have many empty cells. In order to avoid this, we quantize the data as follows:
When a new event is parsed (e.g. at time 5015) a window is opened for 5 ms (configurable) and all events received thereafter are added to the same frame. This is the job of the quantize() function.
See how the objects at the timestamps above are packed into dictionaries below. Each dictionary is a "frame". The original timestamp is shown for each frame. It corresponds to the time of the first event in the XIO file that was added to this frame, i.e. the start of the window. But, in reality, some of the field values might have been logged up to 5 ms later.
As a result, the timestamps of the individual events are "quantized" into the times of the frames. When writing this data back to an XIO file, the events will have these new timestamps instead of their original ones.
In [22]:
list(quantize(f.xio_quicklinegen(5000, 5300), 'VeniceHubReplay/Venice/Body1'))
Out[22]:
inc_recos are special files that are useful within the Incremental Unit (IU) framework.
They store information about units at different update times. Automatic Speech Recognition (ASR) results that are output incrementally can be stored in these files. Each update time is accompanied by a "chunk" of the output (the output at that time).
Here is what they look like:
In [23]:
with open('sampledata/test.inc_reco') as f:
lines = 0
while lines < 22:
print f.readline()[:-1]
lines += 1
But the IncReco class handles these files nicely, e.g.
In [24]:
myincreco = IncReco("sampledata/test.inc_reco")
In [25]:
#get all the update times
print myincreco.get_times()
In [26]:
#get the latest chunk at a specific time
myincreco.get_latest_chunk(5)
Out[26]:
In [27]:
#get the very last chunk -> final output
myincreco.get_last_chunk()
Out[27]:
In addition, you can import an IncReco such as the above as a dictionary of IntervalFrames (one for each chunk):
In [28]:
myrecodict = open_intervalframe_from_increco('sampledata/test.inc_reco')
print myrecodict.keys()
In [29]:
#display the final output intervalframe
myrecodict['5.44']
Out[29]: