o3d3xx-oem-jitter

The o3d3xx-oem-jitter tool provides us with a way to take a pragmatic look at the expected latency and associated jitter that we will encounter when using the framegrabber supplied in the oem module of libo3d3xx. We note that the actual jitter you will see will be heavily dependent upon both your camera/imager settings and your libo3d3xx library settings (most signifcantly your schema mask passed to the o3d3xx::oem::FrameGrabber ctor). There is, of course, also sources of systemic jitter over which we have no control.

Below, we walk through a quick analysis. For our analysis, we employ the single frequency, single exposure imager settings and we set our schema mask to 8 -- i.e., we are only interested in the point cloud data from the camera (i.e., we ignore the amplitude, radial distance, normalized amplitude, etc.). Also, for each data set, we collect timing metrics for 1000 captured frames.

All of the data in our analysis below is available in the data subdirectory of this directory. Additionally, we provide a full o3d3xx-dump of the associated camera settings that were used when we collected a particular data set. The o3d3xx-oem-jitter tool loops (in our case) 1000 times, computing the elapsed time it takes to capture a frame (including parsing the pixel data and populating the data structures integral to the o3d3xx::oem::ImageBuffer).

Here is the help output for o3d3xx-oem-jitter:

$ o3d3xx-oem-jitter --help
o3d3xx OEM Frame Grabber Jitter:

General:
  -v [ --version ]             Print version string and exit
  -h [ --help ]                Produce help message

Connection Information:
  --ip arg (=127.0.0.1)        IP address of the camera
  --xmlrpc-port arg (=80)      XMLRPC port of the camera
  --password arg               Password for configuring the camera

Jitter:
  -f [ --nframes ] arg (=100)  Number of frames to capture
  -o [ --outfile ] arg (=-)    Raw data output file, if not specified, nothing 
                               is written

An example of the output from running o3d3xx-oem-jitter using our invariant settings discussed above, plus, a FrameRate of 5 and an ExposureTime of 5000 usec yields the following output:

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-5000-5-8.txt
Mean:   199.993 ms
Median: 199.996 ms
Stddev: 0.390337 ms
MAD:    0.200394 ms
Raw data has been written to: /tmp/jitter-5000-5-8.txt

While o3d3xx-oem-jitter outputs the Mean, Median, Standard Deviation, and the Median Absolute Error (MAD), the summary statistics we are most interested in are the Median and the MAD. Why we like these statistics is best described in this whitepaper. We also note that o3d3xx-oem-jitter does not factor in a normalizing consistency constant (or rather uses the constant 1) in its MAD computation.

Using the above example, and interpreting our Median and MAD, we make the following statement about the jitter using the above setting: "the most typical deviation from the most typical latency of 199.996 ms will be .200394 ms". We now graphically look at these data.



In [1]:

    
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt

import numpy as np

def plot_raw(infile):
    csv = np.genfromtxt(infile, delimiter='\n')
    plt.plot(np.arange(len(csv)), csv, 'b+')
    plt.grid(True)
    plt.xlabel("Sample Number")
    plt.ylabel("Latency (ms)")
    return csv

def plot_pct(csv):
    plt.plot(np.arange(101), map(lambda x: np.percentile(csv, x),np.arange(101)), 'rx')
    plt.grid(True)
    plt.xlabel("% of sample")
    plt.ylabel("latency (ms)")



In [2]:

    
csv = plot_raw('data/jitter-5000-5-8.txt')



In [3]:

    
plot_pct(csv)

The plot in blue above (the return from plot_raw) is showing us the raw collected data. The red plot above, is a percentile plot. Basically, it allows us to answer the question: What is the worst latency to expect in N% of the cases. To do that, you look along the x-axis to pick N. Then, go up to the plotted value on the y-axis to get the answer.

We now produce these same plots for a few other imager configurations.

ExposureTime: 5000 usec

FrameRate: 20

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-5000-20-8.txt
Mean:   49.9892 ms
Median: 49.9936 ms
Stddev: 0.466445 ms
MAD:    0.164476 ms
Raw data has been written to: /tmp/jitter-5000-20-8.txt



In [4]:

    
csv = plot_raw('data/jitter-5000-20-8.txt')



In [5]:

    
plot_pct(csv)

ExposureTime: 4000 usec

FrameRate: 25

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-4000-25-8.txt
Mean:   40.0259 ms
Median: 39.9323 ms
Stddev: 2.96473 ms
MAD:    1.29371 ms
Raw data has been written to: /tmp/jitter-4000-25-8.txt



In [6]:

    
csv = plot_raw('data/jitter-4000-25-8.txt')



In [7]:

    
plot_pct(csv)

ExposureTime: 3000 usec

FrameRate: 30

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-3000-30-8.txt
Mean:   41.2766 ms
Median: 37.9767 ms
Stddev: 8.23983 ms
MAD:    2.32939 ms
Raw data has been written to: /tmp/jitter-3000-30-8.txt



In [8]:

    
csv = plot_raw('data/jitter-3000-30-8.txt')



In [9]:

    
plot_pct(csv)

ExposureTime: 2000 usec

FrameRate: 30

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-2000-30-8.txt
Mean:   37.9445 ms
Median: 36.9106 ms
Stddev: 7.42972 ms
MAD:    3.37425 ms
Raw data has been written to: /tmp/jitter-2000-30-8.txt



In [10]:

    
csv = plot_raw('data/jitter-2000-30-8.txt')



In [11]:

    
plot_pct(csv)

By looking at the plots above, with our invariant settings of single frequency, single exposure, point cloud (and confidence image) parsing only, it appears as though our most stable ExposureTime and FrameRate is 5000 usecs and 20 fps respectively. To test this hypothesis, we run o3d3xx-oem-jitter three more times with these settings then visualize the results.

Run A:

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-5000-20-8-A.txt
Mean:   49.9925 ms
Median: 50.0117 ms
Stddev: 0.615339 ms
MAD:    0.224411 ms
Raw data has been written to: /tmp/jitter-5000-20-8-A.txt

Run B:

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-5000-20-8-B.txt
Mean:   49.987 ms
Median: 50.0035 ms
Stddev: 0.663366 ms
MAD:    0.227116 ms
Raw data has been written to: /tmp/jitter-5000-20-8-B.txt

Run C:

$ O3D3XX_MASK=8 o3d3xx-oem-jitter -f 1000 -o /tmp/jitter-5000-20-8-C.txt
Mean:   49.9861 ms
Median: 49.9984 ms
Stddev: 0.713794 ms
MAD:    0.214067 ms
Raw data has been written to: /tmp/jitter-5000-20-8-C.txt



In [12]:

    
csvA = np.genfromtxt('data/jitter-5000-20-8-A.txt', delimiter='\n')
csvB = np.genfromtxt('data/jitter-5000-20-8-B.txt', delimiter='\n')
csvC = np.genfromtxt('data/jitter-5000-20-8-C.txt', delimiter='\n')

plt.plot(np.arange(len(csvA)), csvA, 'r+')
plt.plot(np.arange(len(csvB)), csvB, 'g+')
plt.plot(np.arange(len(csvC)), csvC, 'b+')
plt.grid(True)
plt.xlabel("Sample Number")
plt.ylabel("Latency (ms)")









    Out[12]:





<matplotlib.text.Text at 0x7f679521d150>



In [13]:

    
pcts = np.arange(101)
plt.plot(pcts, map(lambda x: np.percentile(csvA, x), pcts), 'rx')
plt.plot(pcts, map(lambda x: np.percentile(csvB, x), pcts), 'gx')
plt.plot(pcts, map(lambda x: np.percentile(csvC, x), pcts), 'bx')
plt.grid(True)
plt.xlabel("% of sample")
plt.ylabel("latency (ms)")









    Out[13]:





<matplotlib.text.Text at 0x7f6795258d90>



In [13]: