Executor API - Executor

A tests executor is a module which supports the execution of a configured set of experiments.

Each experiment is composed by:

  • a target configuration
  • a workload to execute

The executor module can be configured to run a set of workloads (wloads) in each different target configuration of a specified set (confs). These wloads and confs can be specified by the "experiments_conf" input dictionary which is described below at Experiments Configuration.

All the results generated by each experiment will be collected in a results folder. The format and content fo the results forlder is detailed in the last cell of Tests execution below.


In [1]:
import logging

from conf import LisaLogging
LisaLogging.setup()


2016-12-08 11:58:22,693 INFO    : root         : Using LISA logging configuration:
2016-12-08 11:58:22,694 INFO    : root         :   /home/vagrant/lisa/logging.conf

In [2]:
import os
import json

from env import TestEnv
from executor import Executor

Target Configuration

The target configuration it's used to describe and configure your test environment. You can find more details in examples/utils/testenv_example.ipynb.


In [3]:
# Setup a target configuration
my_target_conf = {
    
    # Target platform and board
    "platform"    : 'linux',
    "board"       : 'juno',
    
    # Target board IP/MAC address
    "host"        : '192.168.0.1',
    
    # Login credentials
    "username"    : 'root',
    "password"    : 'juno',

}

Experiments Configuration

The experiments configuration defines the software setups that we need on our hardware target.
This can be given as an argument to an Executor instance or to a TestEnv one.

Elements of the experiments configuration:

  • confs: mandatory platform configurations to be tested.
    • tag: relevant string to identify your configuration.
    • flags: ftrace (to enable ftrace events) is the only one supported at the moment.
    • sched_features: features to be added to /sys/kernel/debug/sched_features.
    • cpufreq: CpuFreq governor and tunables.
    • cgroups: CGroups configuration (controller). The default CGroup will be used otherwise.
  • wloads: mandatory workloads to run on each platform configuration.
  • iterations: number of iterations for each workload.
  • tools: binary tools (available under ./tools/$ARCH/) to install by default; these will be merged with the ones in the target configuration.
  • ftrace: FTrace events to collect for all the experiments configurations which have the "ftrace" flag enabled.
  • modules: modules required by the experiments resulted from the experiments configurations.
  • exclude_modules - modules to be disabled.
  • results_dir: results directory - experiments configuration results directory overrides target one.

In [4]:
my_experiments_conf = {

    # Folder where all the results will be collected
    "results_dir" : "ExecutorExample",

    # Platform configurations to test: you can specify any number of configurations
    "confs" : [
        {  
            "tag"            : "base",                           # Relevant string to identify configuration
            "flags"          : ["ftrace", "freeze_userspace"],   # Enable FTrace events, freeze userspace while running            
            "sched_features" : "NO_ENERGY_AWARE",                # Disable EAS
            "cpufreq"        : {                                 # Use PERFORMANCE CpuFreq
                "governor" : "performance",
            },
        },
        {
            "tag"            : "eas",                            # Relevant string to identify configuration
            "flags"          : ["ftrace", "freeze_userspace"],   # Enable FTrace events, freeze userspace while running
            "sched_features" : "ENERGY_AWARE",                   # Enable EAS
            "cpufreq"        : {                                 # Use PERFORMANCE CpuFreq
                "governor" : "performance",
            },
        },
    ],
    
    # Workloads to run (on each platform configuration)
    "wloads" : {
        # Run hackbench with 1 group using pipes
        "perf" : {
            "type" : "perf_bench",
            "conf" : {
                "class" : "messaging",
                "params" : {
                    "group" :    1,
                    "loop"  :   10,
                    "pipe"  : True,
                    "thread": True,
                }
            }
        },
        # Run a 20% duty-cycle periodic task
        "rta" : {
            "type" : "rt-app",
            "loadref" : "big",
            "conf" : {
                "class"  : "profile",
                "params"  : {
                    "p20" : {
                        "kind"   : "Periodic",
                        "params" : {
                            "duty_cycle_pct" : 20,
                         },
                    },
                },
            },
        },
    },
    
    # Number of iterations for each workloaditerations
    "iterations" : 1,
}
    
my_test_conf = {
    # FTrace events to collect for all the tests configuration which have
    # the "ftrace" flag enabled
    "ftrace"  : {
         "events" : [
            "sched_switch",
            "sched_wakeup",
            "sched_wakeup_new",
            "cpu_frequency",
         ],
         "buffsize" : 80 * 1024,
    },
    
    # Tools required by the experiments
    "tools"   : [ 'trace-cmd', 'perf' ],
    
    # Modules required by these experiments
    "modules"     : [ 'bl', 'cpufreq', 'cgroups' ],

}

Tests execution


In [5]:
executor = Executor(TestEnv(target_conf=my_target_conf, test_conf=my_test_conf), my_experiments_conf)


2016-12-07 10:17:28,037 INFO    : TestEnv      : Using base path: /home/vagrant/lisa
2016-12-07 10:17:28,039 INFO    : TestEnv      : Loading custom (inline) target configuration
2016-12-07 10:17:28,039 INFO    : TestEnv      : Devlib modules to load: ['bl', 'hwmon', 'cpufreq']
2016-12-07 10:17:28,040 INFO    : TestEnv      : Connecting linux target:
2016-12-07 10:17:28,040 INFO    : TestEnv      :   username : root
2016-12-07 10:17:28,041 INFO    : TestEnv      :       host : 192.168.0.1
2016-12-07 10:17:28,041 INFO    : TestEnv      :   password : juno
2016-12-07 10:17:28,041 INFO    : TestEnv      : Connection settings:
2016-12-07 10:17:28,042 INFO    : TestEnv      :    {'username': 'root', 'host': '192.168.0.1', 'password': 'juno'}
2016-12-07 10:17:45,282 INFO    : TestEnv      : Initializing target workdir:
2016-12-07 10:17:45,283 INFO    : TestEnv      :    /root/devlib-target
2016-12-07 10:17:51,006 INFO    : TestEnv      : Topology:
2016-12-07 10:17:51,007 INFO    : TestEnv      :    [[0, 3, 4, 5], [1, 2]]
2016-12-07 10:17:52,248 INFO    : EnergyMeter  : Scanning for HWMON channels, may take some time...
2016-12-07 10:17:52,250 INFO    : EnergyMeter  : Channels selected for energy sampling:
2016-12-07 10:17:52,250 INFO    : EnergyMeter  :    BOARDBIG_energy
2016-12-07 10:17:52,251 INFO    : EnergyMeter  :    BOARDLITTLE_energy
2016-12-07 10:17:52,251 INFO    : TestEnv      : Set results folder to:
2016-12-07 10:17:52,251 INFO    : TestEnv      :    /home/vagrant/lisa/results/20161207_101752
2016-12-07 10:17:52,252 INFO    : TestEnv      : Experiment results available also in:
2016-12-07 10:17:52,252 INFO    : TestEnv      :    /home/vagrant/lisa/results_latest
2016-12-07 10:17:52,253 INFO    : Executor     : Loading custom (inline) test configuration
2016-12-07 10:17:52,253 INFO    : Executor     : 
2016-12-07 10:17:52,254 INFO    : Executor     : ################################################################################
2016-12-07 10:17:52,254 INFO    : Executor     : Experiments configuration
2016-12-07 10:17:52,254 INFO    : Executor     : ################################################################################
2016-12-07 10:17:52,255 INFO    : Executor     : Configured to run:
2016-12-07 10:17:52,255 INFO    : Executor     :      2 target configurations:
2016-12-07 10:17:52,256 INFO    : Executor     :       base, eas
2016-12-07 10:17:52,256 INFO    : Executor     :      2 workloads (1 iterations each)
2016-12-07 10:17:52,257 INFO    : Executor     :       rta, perf
2016-12-07 10:17:52,257 INFO    : Executor     : Total: 4 experiments
2016-12-07 10:17:52,257 INFO    : Executor     : Results will be collected under:
2016-12-07 10:17:52,258 INFO    : Executor     :       /home/vagrant/lisa/results/20161207_101752

In [6]:
executor.run()


2016-12-07 10:17:59,239 INFO    : Executor     : 
2016-12-07 10:17:59,239 INFO    : Executor     : ################################################################################
2016-12-07 10:17:59,240 INFO    : Executor     : Experiments execution
2016-12-07 10:17:59,240 INFO    : Executor     : ################################################################################
2016-12-07 10:17:59,241 INFO    : Executor     : 
2016-12-07 10:17:59,241 INFO    : Executor     : ================================================================================
2016-12-07 10:17:59,241 INFO    : Executor     : configuring target for [base] experiments
2016-12-07 10:18:00,663 INFO    : Executor     : Set scheduler feature: NO_ENERGY_AWARE
2016-12-07 10:18:01,469 INFO    : Executor     : Configuring all CPUs to use [performance] cpufreq governor
2016-12-07 10:18:02,274 INFO    : Workload     : Setup new workload rta
2016-12-07 10:18:02,274 INFO    : Workload     : Workload duration defined by longest task
2016-12-07 10:18:02,275 INFO    : Workload     : Default policy: SCHED_OTHER
2016-12-07 10:18:02,275 INFO    : Workload     : ------------------------
2016-12-07 10:18:02,276 INFO    : Workload     : task [task_p200], sched: using default policy
2016-12-07 10:18:02,276 INFO    : Workload     :  | calibration CPU: 1
2016-12-07 10:18:02,277 INFO    : Workload     :  | loops count: 1
2016-12-07 10:18:02,277 INFO    : Workload     : + phase_000001: duration 1.000000 [s] (10 loops)
2016-12-07 10:18:02,278 INFO    : Workload     : |  period   100000 [us], duty_cycle  20 %
2016-12-07 10:18:02,278 INFO    : Workload     : |  run_time  20000 [us], sleep_time  80000 [us]
2016-12-07 10:18:05,732 INFO    : Executor     : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-12-07 10:18:05,734 INFO    : Executor     : Experiment 0/4, [base:rta] 1/1
2016-12-07 10:18:06,361 INFO    : Workload     : Workload execution START:
2016-12-07 10:18:06,363 INFO    : Workload     :    /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1
2016-12-07 10:18:13,384 INFO    : Executor     : --------------------------------------------------------------------------------
2016-12-07 10:18:13,386 INFO    : Workload     : Setup new workload perf
2016-12-07 10:18:13,712 INFO    : Executor     : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-12-07 10:18:13,712 INFO    : Executor     : Experiment 1/4, [base:perf] 1/1
2016-12-07 10:18:14,337 INFO    : Workload     : Workload execution START:
2016-12-07 10:18:14,338 INFO    : Workload     :    /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10
2016-12-07 10:18:14,664 INFO    : perf_bench   :      PerfBench - Completion time: 0.010000, Performance 100.000000
2016-12-07 10:18:15,286 INFO    : Executor     : --------------------------------------------------------------------------------
2016-12-07 10:18:15,288 INFO    : Executor     : 
2016-12-07 10:18:15,290 INFO    : Executor     : ================================================================================
2016-12-07 10:18:15,292 INFO    : Executor     : configuring target for [eas] experiments
2016-12-07 10:18:16,713 INFO    : Executor     : Set scheduler feature: ENERGY_AWARE
2016-12-07 10:18:17,519 INFO    : Executor     : Configuring all CPUs to use [performance] cpufreq governor
2016-12-07 10:18:18,325 INFO    : Workload     : Setup new workload rta
2016-12-07 10:18:18,326 INFO    : Workload     : Workload duration defined by longest task
2016-12-07 10:18:18,327 INFO    : Workload     : Default policy: SCHED_OTHER
2016-12-07 10:18:18,329 INFO    : Workload     : ------------------------
2016-12-07 10:18:18,330 INFO    : Workload     : task [task_p200], sched: using default policy
2016-12-07 10:18:18,331 INFO    : Workload     :  | calibration CPU: 1
2016-12-07 10:18:18,332 INFO    : Workload     :  | loops count: 1
2016-12-07 10:18:18,332 INFO    : Workload     : + phase_000001: duration 1.000000 [s] (10 loops)
2016-12-07 10:18:18,333 INFO    : Workload     : |  period   100000 [us], duty_cycle  20 %
2016-12-07 10:18:18,333 INFO    : Workload     : |  run_time  20000 [us], sleep_time  80000 [us]
2016-12-07 10:18:21,414 INFO    : Executor     : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-12-07 10:18:21,415 INFO    : Executor     : Experiment 2/4, [eas:rta] 1/1
2016-12-07 10:18:22,043 INFO    : Workload     : Workload execution START:
2016-12-07 10:18:22,045 INFO    : Workload     :    /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1
2016-12-07 10:18:28,802 INFO    : Executor     : --------------------------------------------------------------------------------
2016-12-07 10:18:28,806 INFO    : Workload     : Setup new workload perf
2016-12-07 10:18:29,134 INFO    : Executor     : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2016-12-07 10:18:29,135 INFO    : Executor     : Experiment 3/4, [eas:perf] 1/1
2016-12-07 10:18:29,760 INFO    : Workload     : Workload execution START:
2016-12-07 10:18:29,761 INFO    : Workload     :    /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10
2016-12-07 10:18:30,089 INFO    : perf_bench   :      PerfBench - Completion time: 0.009000, Performance 111.111111
2016-12-07 10:18:30,710 INFO    : Executor     : --------------------------------------------------------------------------------
2016-12-07 10:18:30,712 INFO    : Executor     : 
2016-12-07 10:18:30,714 INFO    : Executor     : ################################################################################
2016-12-07 10:18:30,715 INFO    : Executor     : Experiments execution completed
2016-12-07 10:18:30,716 INFO    : Executor     : ################################################################################
2016-12-07 10:18:30,716 INFO    : Executor     : Results available in:
2016-12-07 10:18:30,717 INFO    : Executor     :       /home/vagrant/lisa/results/20161207_101752

In [7]:
!tree {executor.te.res_dir}


/home/vagrant/lisa/results/20161207_101752
├── perf_bench_messaging:base:perf
│   ├── 1
│   │   ├── energy.json
│   │   ├── output.log
│   │   └── performance.json
│   ├── kernel.config
│   ├── kernel.version
│   └── platform.json
├── perf_bench_messaging:eas:perf
│   ├── 1
│   │   ├── energy.json
│   │   ├── output.log
│   │   └── performance.json
│   ├── kernel.config
│   ├── kernel.version
│   └── platform.json
├── rtapp:base:rta
│   ├── 1
│   │   ├── energy.json
│   │   ├── output.log
│   │   ├── rta_00.json
│   │   └── rt-app-task_p200-0.log
│   ├── kernel.config
│   ├── kernel.version
│   └── platform.json
└── rtapp:eas:rta
    ├── 1
    │   ├── energy.json
    │   ├── output.log
    │   ├── rta_00.json
    │   └── rt-app-task_p200-0.log
    ├── kernel.config
    ├── kernel.version
    └── platform.json

8 directories, 26 files