Frequency_Invariance_Test


Frequency Invariant Load Tracking Test

FreqInvarianceTest is a LisaTest class for automated testing of frequency invariant load tracking. This notebook uses the methods it provides to perform the same analysis as the automated test and plot some results.

The test class runs the same workload at a selection of frequencies, each entry in t.experiments represents a run at a different frequency.

Setup


In [1]:
%matplotlib inline

In [2]:
import pandas as pd
import json

from trace import Trace
from trappy.plotter import plot_trace
from trappy.stats.grammar import Parser
from trappy import ILinePlot

In [3]:
import logging
from conf import LisaLogging
LisaLogging.setup()
logging.getLogger('Analysis').setLevel(logging.ERROR)
logging.getLogger('Trace').setLevel(logging.ERROR)


2017-02-13 18:36:59,553 INFO    : root         : Using LISA logging configuration:
2017-02-13 18:36:59,554 INFO    : root         :   /home/brendan/sources/lisa/logging.conf

Run test workload

There's currently no way to pass a TestEnv or configuration to automated test classes. Instead the target information comes from the target.config file (in the root of the LISA source tree), so you'll need to edit that to configure LISA to connect to your target.


In [4]:
from tests.eas.load_tracking import FreqInvarianceTest

t = FreqInvarianceTest()
print t.__doc__


    Goal
    ====
    Basic check for frequency invariant load tracking

    Detailed Description
    ====================
    This test runs the same workload on the most capable CPU on the system at a
    cross section of available frequencies. The trace is then examined to find
    the average activation length of the workload, which is combined with the
    known period to estimate an expected mean value for util_avg for each
    frequency. The util_avg value is extracted from scheduler trace events and
    its mean is compared with the expected value (ignoring the first 300ms so
    that the signal can stabilize). The test fails if the observed mean is
    beyond a certain error margin from the expected one. load_avg is then
    similarly compared with the expected util_avg mean, under the assumption
    that load_avg should equal util_avg when system load is light.

    Expected Behaviour
    ==================
    Load tracking signals are scaled so that the workload results in roughly the
    same util & load values regardless of frequency.
    

To run automated tests from within a notebook we instantiate the test class and call runExperiments on it.


In [5]:
t.runExperiments()


2017-02-13 18:36:59,608 INFO    : LisaTest     : Setup tests execution engine...
2017-02-13 18:36:59,609 INFO    : TestEnv      : Using base path: /home/brejac01/sources/lisa
2017-02-13 18:36:59,610 INFO    : TestEnv      : Loading default (file) target configuration
2017-02-13 18:36:59,611 INFO    : TestEnv      : Loading target configuration [/home/brejac01/sources/lisa/target.config]...
2017-02-13 18:36:59,613 INFO    : TestEnv      : Loading custom (inline) test configuration
2017-02-13 18:36:59,614 INFO    : TestEnv      : Devlib modules to load: ['bl', u'cpuidle', 'cgroups', 'cpufreq']
2017-02-13 18:36:59,615 INFO    : TestEnv      : Connecting linux target:
2017-02-13 18:36:59,616 INFO    : TestEnv      :   username : brendan
2017-02-13 18:36:59,617 INFO    : TestEnv      :       host : 192.168.2.2
2017-02-13 18:36:59,618 INFO    : TestEnv      :   password : password
2017-02-13 18:36:59,618 INFO    : TestEnv      : Connection settings:
2017-02-13 18:36:59,619 INFO    : TestEnv      :    {'username': u'brendan', 'host': u'192.168.2.2', 'password': u'password'}
2017-02-13 18:37:07,367 INFO    : TestEnv      : Initializing target workdir:
2017-02-13 18:37:07,369 INFO    : TestEnv      :    /home/brendan/devlib-target
2017-02-13 18:37:11,034 INFO    : CGroups      : Available controllers:
2017-02-13 18:37:12,775 INFO    : CGroups      :   cpuset       : /home/brendan/devlib-target/cgroups/devlib_cgh2
2017-02-13 18:37:14,512 INFO    : CGroups      :   cpu          : /home/brendan/devlib-target/cgroups/devlib_cgh3
2017-02-13 18:37:16,249 INFO    : CGroups      :   cpuacct      : /home/brendan/devlib-target/cgroups/devlib_cgh3
2017-02-13 18:37:17,988 INFO    : CGroups      :   blkio        : /home/brendan/devlib-target/cgroups/devlib_cgh4
2017-02-13 18:37:19,726 INFO    : CGroups      :   memory       : /home/brendan/devlib-target/cgroups/devlib_cgh5
2017-02-13 18:37:21,465 INFO    : CGroups      :   devices      : /home/brendan/devlib-target/cgroups/devlib_cgh6
2017-02-13 18:37:23,205 INFO    : CGroups      :   perf_event   : /home/brendan/devlib-target/cgroups/devlib_cgh7
2017-02-13 18:37:24,944 INFO    : CGroups      :   hugetlb      : /home/brendan/devlib-target/cgroups/devlib_cgh8
2017-02-13 18:37:26,682 INFO    : CGroups      :   pids         : /home/brendan/devlib-target/cgroups/devlib_cgh9
2017-02-13 18:37:29,594 INFO    : TestEnv      : Topology:
2017-02-13 18:37:29,596 INFO    : TestEnv      :    [[0, 3, 4, 5], [1, 2]]
2017-02-13 18:37:32,227 INFO    : TestEnv      : Loading default EM:
2017-02-13 18:37:32,229 INFO    : TestEnv      :    /home/brejac01/sources/lisa/libs/utils/platforms/juno.json
2017-02-13 18:37:33,996 WARNING : LinuxTarget  : Event [sched_load_avg_task] not available for tracing
2017-02-13 18:37:33,999 WARNING : LinuxTarget  : Event [sched_load_avg_cpu] not available for tracing
2017-02-13 18:37:34,001 INFO    : TestEnv      : Enabled tracepoints:
2017-02-13 18:37:34,002 INFO    : TestEnv      :    sched_switch
2017-02-13 18:37:34,003 INFO    : TestEnv      :    sched_load_avg_task
2017-02-13 18:37:34,004 INFO    : TestEnv      :    sched_load_avg_cpu
2017-02-13 18:37:34,006 INFO    : TestEnv      :    sched_pelt_se
2017-02-13 18:37:34,007 WARNING : TestEnv      : Using configuration provided RTApp calibration
2017-02-13 18:37:34,008 INFO    : TestEnv      : Using RT-App calibration values:
2017-02-13 18:37:34,010 INFO    : TestEnv      :    {"0": 354, "1": 138, "2": 138, "3": 363, "4": 355, "5": 357}
2017-02-13 18:37:34,011 INFO    : EnergyMeter  : HWMON module not enabled
2017-02-13 18:37:34,012 WARNING : EnergyMeter  : Energy sampling disabled by configuration
2017-02-13 18:37:34,013 INFO    : TestEnv      : Set results folder to:
2017-02-13 18:37:34,015 INFO    : TestEnv      :    /home/brejac01/sources/lisa/results/20170213_183734
2017-02-13 18:37:34,016 INFO    : TestEnv      : Experiment results available also in:
2017-02-13 18:37:34,017 INFO    : TestEnv      :    /home/brejac01/sources/lisa/results_latest
2017-02-13 18:37:34,018 INFO    : Executor     : Loading custom (inline) test configuration
2017-02-13 18:37:34,019 INFO    : Executor     : 
2017-02-13 18:37:34,020 INFO    : Executor     : ################################################################################
2017-02-13 18:37:34,022 INFO    : Executor     : Experiments configuration
2017-02-13 18:37:34,023 INFO    : Executor     : ################################################################################
2017-02-13 18:37:34,024 INFO    : Executor     : Configured to run:
2017-02-13 18:37:34,025 INFO    : Executor     :      1 target configurations:
2017-02-13 18:37:34,026 INFO    : Executor     :       freq_450000
2017-02-13 18:37:34,027 INFO    : Executor     :      1 workloads (1 iterations each)
2017-02-13 18:37:34,028 INFO    : Executor     :       fie_10pct
2017-02-13 18:37:34,029 INFO    : Executor     : Total: 1 experiments
2017-02-13 18:37:34,030 INFO    : Executor     : Results will be collected under:
2017-02-13 18:37:34,031 INFO    : Executor     :       /home/brejac01/sources/lisa/results/20170213_183734
2017-02-13 18:37:34,032 INFO    : Executor     : rt-app workloads found, installing tool on target
2017-02-13 18:37:34,033 INFO    : LisaTest     : Experiments execution...
2017-02-13 18:37:34,034 INFO    : Executor     : 
2017-02-13 18:37:34,035 INFO    : Executor     : ################################################################################
2017-02-13 18:37:34,036 INFO    : Executor     : Experiments execution
2017-02-13 18:37:34,037 INFO    : Executor     : ################################################################################
2017-02-13 18:37:34,038 INFO    : Executor     : 
2017-02-13 18:37:34,039 INFO    : Executor     : ================================================================================
2017-02-13 18:37:34,040 INFO    : Executor     : configuring target for [freq_450000] experiments
2017-02-13 18:37:35,760 INFO    : Executor     : Configuring all CPUs to use [userspace] cpufreq governor
2017-02-13 18:37:36,666 INFO    : Executor     :        CPUFreq - CPU frequencies: {1: 450000}
2017-02-13 18:37:38,683 INFO    : Workload     : Setup new workload fie_10pct
2017-02-13 18:37:38,685 INFO    : Workload     : Workload duration defined by longest task
2017-02-13 18:37:38,686 INFO    : Workload     : Default policy: SCHED_OTHER
2017-02-13 18:37:38,687 INFO    : Workload     : ------------------------
2017-02-13 18:37:38,688 INFO    : Workload     : task [fie_test0], sched: using default policy
2017-02-13 18:37:38,689 INFO    : Workload     :  | calibration CPU: 1
2017-02-13 18:37:38,691 INFO    : Workload     :  | loops count: 1
2017-02-13 18:37:38,692 INFO    : Workload     : + phase_000001: duration 1.000000 [s] (62 loops)
2017-02-13 18:37:38,693 INFO    : Workload     : |  period    16000 [us], duty_cycle  10 %
2017-02-13 18:37:38,694 INFO    : Workload     : |  run_time   1600 [us], sleep_time  14400 [us]
2017-02-13 18:37:39,836 INFO    : Executor     : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2017-02-13 18:37:39,838 INFO    : Executor     : Experiment 0/1, [freq_450000:fie_10pct] 1/1
2017-02-13 18:37:40,255 WARNING : Executor     : No freezer cgroup controller on target. Not freezing userspace
2017-02-13 18:37:40,257 WARNING : Executor     : FTrace events collection enabled
2017-02-13 18:37:46,898 INFO    : Workload     : Workload execution START:
2017-02-13 18:37:46,899 INFO    : Workload     :    /home/brendan/devlib-target/bin/taskset 0x2 /home/brendan/devlib-target/bin/rt-app /home/brendan/devlib-target/run_dir/fie_10pct_00.json 2>&1

2017-02-13 18:37:54,459 INFO    : Executor     : Collected FTrace binary trace:
2017-02-13 18:37:54,460 INFO    : Executor     :    <res_dir>/rtapp:freq_450000:fie_10pct/1/trace.dat
2017-02-13 18:37:54,461 INFO    : Executor     : Collected FTrace function profiling:
2017-02-13 18:37:54,463 INFO    : Executor     :    <res_dir>/rtapp:freq_450000:fie_10pct/1/trace_stat.json
2017-02-13 18:37:54,463 INFO    : Executor     : --------------------------------------------------------------------------------
2017-02-13 18:37:54,464 INFO    : Executor     : 
2017-02-13 18:37:54,465 INFO    : Executor     : ################################################################################
2017-02-13 18:37:54,467 INFO    : Executor     : Experiments execution completed
2017-02-13 18:37:54,467 INFO    : Executor     : ################################################################################
2017-02-13 18:37:54,468 INFO    : Executor     : Results available in:
2017-02-13 18:37:54,469 INFO    : Executor     :       /home/brejac01/sources/lisa/results/20170213_183734

Show variance in util_avg and load_avg

We want to see the same util_avg and load_avg values regardless of frequencies - the bar charts below should have bars all with roughly the same height.


In [6]:
# Get the frequency an experiment was run at
def experiment_freq(exp):
    [cpu] = exp.wload.cpus
    freq = exp.conf['cpufreq']['freqs'][cpu]
    return freq
freqs = [experiment_freq(e) for e in t.experiments]

In [7]:
def plot_signal_against_freq(signal):
    means = [t.get_signal_mean(e, signal) for e in t.experiments]
    limits = (0 , max(means) * 1.15)
    pd.DataFrame(means, index=freqs, columns=['Mean ' + signal]).plot(kind='bar', ylim=limits)

Plot of variation of util_avg value with frequency:


In [8]:
t.get_trace(t.experiments[0]).available_events


Out[8]:
['sched_switch', 'sched_pelt_se']

In [9]:
plot_signal_against_freq('util_avg')


And the same thing for load_avg:


In [10]:
plot_signal_against_freq('load_avg')


Examine trace from workload execution

Plot task residency and sched_util and sched_load for the workload task, along with the expected mean value for util_avg. Note that assuming the system was under little load, so that the task was RUNNING whenever it was RUNNABLE, load_avg and util_avg should be the same.

Call examine_experiment with different experiment indexes to get plots for runs at different frequencies.


In [11]:
signals = ['util_avg', 'load_avg']
def examine_experiment(idx):
    experiment = t.experiments[idx]
    
    [freq] = experiment.conf['cpufreq']['freqs'].values()
    print "Experiment ran at frequency {}".format(freq)
    events = t.te.test_conf["ftrace"]["events"]
    
    print 'Trace plot:'
    plot_trace(t.get_trace(experiment).ftrace)
    
    # Get observed signal
    signal_df = t.get_sched_task_signals(experiment, signals)
    # Get expected average value for util_avg signal
    expected_util_avg_mean = t.get_expected_util_avg(experiment)
    
    # Plot task util_avg signal with expected mean value
    util_avg_mean = pd.Series([expected_util_avg_mean], name='expected_util_avg', index=[signal_df.index[0]])
    df = pd.concat([signal_df, util_avg_mean], axis=1).ffill()
    ILinePlot(df, column=signals + ['expected_util_avg'], drawstyle='steps-post',
             title='Scheduler task signals').view()

In [12]:
for i , f in enumerate(freqs):
    print "Experiment {}:{:10d}Hz".format(i, f)


Experiment 0:    450000Hz

In [13]:
examine_experiment(0)


Experiment ran at frequency 450000
Trace plot:
Scheduler task signals