Mumodo Demo Notebook -- Updated on 24.04.2015
Summary: This notebook explains how to synchronize data that comes from different sensors. There is tracking data stored in an XIO file (see relevant notebook) and Audio-Video data. The method described uses a visible Timecode in the video that represents the simultaneous timestamp of the tracking data
(c) Dialogue Systems Group, University of Bielefeld
In [1]:
%matplotlib inline
from IPython.display import Image
from mumodo.xiofile import XIOFile
from mumodo.mumodoIO import open_streamframe_from_xiofile
from mumodo.plotting import plot_scalar
import pandas as pd
After recording a video and audio with a camera and tracking with venice.hub, we need to synchronize the two data channels. We use a visual synchronization method, by displaying the timestamp of the computer logging data with venice.hub in the view of the video camera.
In [2]:
Image(filename="sampledata/testimage.png")
Out[2]:
We load the video in an editor with frame-by-frame seeking capabilities and find a frame in the video in which the numbers on the screen can be read well (such as above). We store this number and its time in the video ((converted to milliseconds), e.g.
In [3]:
Timepoints = []
Timepoints.append({'video_time': 2408, 'timestamp': 9192636608})
Timepoints
Out[3]:
We now turn to our recorded tracking data, of which we want to find the minimum time, i.e. the *first timestamp in the XIO file", as follows:
In [4]:
minXIOtime = XIOFile("sampledata/test.xio.gz").min_time
minXIOtime
Out[4]:
We see that the last 10 digits of the timestamp were visible on the screem, so we keep only those for our minXIOtime:
In [5]:
minXIOtime = minXIOtime % 10000000000
minXIOtime
Out[5]:
Next, we adjust the timestamps in Timepoints to be relative to this initial timestamp, e.g.
In [6]:
for tp in Timepoints:
tp['timestamp'] -= minXIOtime
Timepoints
Out[6]:
Finally, we compute the offset between this two times: The timestamp is relative to the beginning of the XIO file, and the 'video time' is relative to the beginning of the video file. Hence, their difference is the offset between these two files
In [7]:
for tp in Timepoints:
tp['offset'] = tp['video_time'] - tp['timestamp']
Timepoints
Out[7]:
So we compute an offset on -9621 ms, which means can be used in venice.hub for synchronized playback of the data, or to analyze the tracking data that we have recorded at times of interest taken from the audio/video (e.g. transcriptions)
It is possible to do this at more than one timepoint to make sure it is done correctly and to be more precise, by computing an average offset, e.g.:
In [8]:
Timepoints = []
Timepoints.append({'video_time': 2408, 'timestamp': 9192636608})
Timepoints.append({'video_time': 4015, 'timestamp': 9192638239})
Timepoints.append({'video_time': 7965, 'timestamp': 9192642139})
Timepoints.append({'video_time': 15170, 'timestamp': 9192649352})
for tp in Timepoints:
tp['timestamp'] -= minXIOtime
tp['offset'] = tp['video_time'] - tp['timestamp']
TimepointsFrame = pd.DataFrame(Timepoints)
TimepointsFrame
Out[8]:
The offset can now be computed more precisely:
In [9]:
TimepointsFrame['offset'].mean(), TimepointsFrame['offset'].std()
Out[9]:
Replaying the data with venice.hub
java - jar venicehub.jar -i Disk -o IIO -f test.xio.gz --offset -9616The following command imports the skeleton data of all skeletons into one mumodo StreamFrame. The arguments are:
In [10]:
skeletons = open_streamframe_from_xiofile("sampledata/test.xio.gz", 'VeniceHubReplay/Venice/Body1', timestamp_offset=-9616)
In [11]:
skeletons.ix[:, ['JointPositions3', 'JointPositions4']].ix[0:100]
Out[11]:
Facelab Eyetracker has an inherent lag due to tracking filters, when logging the 'accurate', rather than the 'realtime' data. That is, the timestamp of the index (time of logging) is not the same as the timestamp of the event described by the data point.
Luckily, we have access to the time of frame capture from the cameras in the data point itself.
The timestamp_offset kwargs allows us to import the raw timestamps, so that we can compute the lag!
In [12]:
SensorStream = open_streamframe_from_xiofile("sampledata/othersensor.xio.gz",
"Sensors/EyeTracker",
timestamp_offset=None)
SensorStream.dropna(inplace=True)
In the table below, the index is the (raw) time when the event was logged
However, frametimeSeconds and frameTimeMilliSeconds jointly store the time when the video frame was captured, that the data is based on
In [13]:
SensorStream[:10]
Out[13]:
In [14]:
#Compute the time of the camera frame capture, from the two relevant fields
SensorStream['time_real'] = 1000 * SensorStream['frameTimeSeconds'] + SensorStream['frameTimeMilliSeconds']
In [15]:
#plot the lag betweem the frame capture and the time of logging
SensorStream['lag'] = SensorStream['time']-SensorStream['time_real']
plot_scalar(SensorStream, ['lag']) #the Pandas f['lag'].plot() can also be used
print 'mean lag: {} \nstd of lag: {}'.format(SensorStream['lag'].mean(), SensorStream['lag'].std())
We see than the mean lag is 1825 ms on average. Re-sync this input and use its time as index. Again, subtract the first timestamp to get relative times
In [16]:
SensorStream.drop_duplicates(subset=['time_real'], inplace=True)
SensorStream.index = SensorStream['time_real'].map(lambda x: int(x) - SensorStream.index[0])
SensorStream = SensorStream.sort_index()
In [17]:
SensorStream[:10]
Out[17]:
As the negative indices now show, the actual time of the event at time zero, actually occurred 1827 ms before that and so on. In other words, the data point at time 6 ms below is the one that actually occurred at time 6, not logged at that time
In [18]:
SensorStream.ix[10:25]
Out[18]:
If we already know the lag, then we can import the file and synhronize the times directly as follows:
In [19]:
SensorStream = open_streamframe_from_xiofile("sampledata/othersensor.xio.gz",
"Sensors/EyeTracker",
timestamp_offset=-1825)
SensorStream.dropna(inplace=True)
In [20]:
SensorStream.ix[0:20]
Out[20]:
This difference of 7 ms from the previous method is due to the fact that in the second method, we synchronized all points with the mean lag. In the first method, we happen to know the lag for each point, hence we can compute the mean lag. It is more common that we only have a rough idea about the lag, so the second method is included in the open_streamframe_from_xiofile() as standard