Mumodo Demo Notebook - Updated on 24.04.2015
Summary: This notebook describes the basic data types used by mumodo. In particular, the IntervalFrame, StreamFrame, as well as the InstantIO types are described.
(c) Dialogue Systems Group, University of Bielefeld
In [1]:
from mumodo.mumodoIO import open_intervalframe_from_textgrid, open_streamframe_from_xiofile, convert_pointtier_to_streamframe
from mumodo.analysis import convert_times_of_tier
from mumodo.InstantIO import *
import pandas as pd
Analysis in mumodo is based on only two generic types of objects. The StreamFrame and the IntervalFrame. Let's have a look at them.
In [2]:
mystreamframe = open_streamframe_from_xiofile("sampledata/test.xio.gz",
'VeniceHubReplay/Venice/Body1').ix[:, ['JointPositions3', 'JointPositions4']]
A StreamFrame is a Pandas DataFrame object (See Pandas Documentation), that satisfies the following conditions
So, in essence, StreamFrames are multivariate time series.
The powerful Pandas functionality allows easy access to specific cells of the table, using the time and the name of the column e.g.
In [3]:
#so values of the first column in the first 1000 ms
print mystreamframe.columns
mystreamframe.ix[0:1000, ['JointPositions3']]
Out[3]:
In [4]:
#print a specific value
x = mystreamframe['JointPositions4'][984]
print x
In [5]:
#print this value's type, these types are explained below
print type(x)
In [6]:
myintervalframe = open_intervalframe_from_textgrid("sampledata/test.TextGrid")['S']
An IntervalFrame is a Pandas DataFrame object (See Pandas Documentation), that satisfies the following conditions
So, in essence, IntervalFrames are annotation tiers commonly found in audio, video and other analysis software (e.g. Praat, ELAN)
The advantages of Pandas also apply to this category of objects, e.g.
In [7]:
myintervalframe
Out[7]:
In [8]:
mypointframe = open_intervalframe_from_textgrid("sampledata/test.TextGrid")['CLAPS']
A PointFrame is a Pandas DataFrame object (See Pandas Documentation), that satisfies the following conditions
PointFrames are very similar to StreamFrames, but sometimes they need to be imported directly from Praat Textgrids (Praat Point Tiers) and are thus more like annotations without duration, rather than raw tracking data.
The advantages of Pandas also apply to this category of objects, e.g.
In [9]:
mypointframe
Out[9]:
It is possible to convert the point_tier to a streamframe (the units are not converted) using a specially designed function
In [10]:
#first convert the units to ms
convert_times_of_tier(mypointframe, lambda x: int(1000 * x))
#next, convert to a streamframe
mynewstreamframe = convert_pointtier_to_streamframe(mypointframe)
mynewstreamframe
Out[10]:
Multimodal data typically consists of audio, video and other sensors, such as motion capture, eye tracking, etc.
In order to deal with this complex data, mumodo supports a number of types inspired by the InstantIO framework for Virtual Reality (www.instantreality.org). The following types are supported
So let's have a look at the value x that we got from our table above ...
In [11]:
type(x)
Out[11]:
It is a multi-field SFVec3f. Is that really so?
In [12]:
type(x[0])
Out[12]:
Indeed! MF types also have length and are iterable
In [13]:
for y in x:
print y
In [14]:
len(x)
Out[14]:
The basic (SF) types have some useful atttibutes, e.g.
In [15]:
my3Dvector = x[0]
print type(my3Dvector)
print my3Dvector.x, my3Dvector.y, my3Dvector.z #per axis x, y, z access
print my3Dvector.v #get it as list
print list(my3Dvector) #iterate
print my3Dvector[2] #array-like access for elements
the rotations have similarly useful functions, e.g.
In [16]:
myrotation = SFRotation(0, 0.5, 0, 0.5)
print myrotation.qx, myrotation.qy, myrotation.qz, myrotation.qw #components of the quaternion
print myrotation.v #get is as a list
print list(myrotation) #iterate
print myrotation[3] #array-like access for elements
print myrotation.attitude, myrotation.bank, myrotation.heading #get as Euler angles in radians
Strings are parsed into boolean by the sfbool function, with a different behaviour from the built-in bool: the string True is case-insensitive parsed into True, and any other string is False
In [17]:
sfbool("False"), sfbool("anything"), sfbool("tRue")
Out[17]:
In [18]:
bool("False"), bool("anything"), bool("tRue")
Out[18]:
What if your own data is not using these types?
Let's say your 3D data looks like below. This is typical if you import your data from CSV files or directly from Excel files, Pandas has functions for that.
In [19]:
mydata = pd.DataFrame([{'2dpositions': [2, 4], 'time': 0},
{'2dpositions': [2, 3], 'time': 10},
{'2dpositions': [3, 3], 'time': 20},
{'2dpositions': [7, 4], 'time': 30},
{'2dpositions': [1, 2], 'time': 40},
{'2dpositions': [0, 4], 'time': 50}])
mydata
Out[19]:
First we need to make this look like a streamframe, so we have to make the time into the index (and optionally delete the time column)
In [20]:
mydata.index = mydata.time #alternative syntax to mydata['time']
mydata.index.name = None #we do not need a named index
del mydata['time']
mydata
Out[20]:
Now we want to perhaps change the type of our 2d positions into SFVec2f.
In [21]:
mydata['2dpositions'] = mydata['2dpositions'].map(lambda x: SFVec2f(x[0], x[1]))
mydata
Out[21]:
And thus we have converted our data into the basic mumodo types. This will be very useful for our subsequent analysis
In order to see how to use IntervalFrames, StreamFrames and InstantIO types for analysis, check the other demo notebooks