cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
A control group is a collection of processes that are bound by the same criteria and associated with a set of parameters or limits. These groups can be hierarchical, meaning that each group inherits limits from its parent group. The kernel provides access to multiple controllers (also called subsystems) through the cgroup interface, for example, the "memory" controller limits memory use, "cpuacct" accounts CPU usage, etc.
In [1]:
import logging
from conf import LisaLogging
LisaLogging.setup()
In [2]:
import os
import json
import operator
import devlib
import trappy
import bart
from bart.sched.SchedMultiAssert import SchedMultiAssert
from wlgen import RTA, Periodic
In [3]:
from env import TestEnv
my_conf = {
# Android Pixel
"platform" : "android",
"board" : "pixel",
"device" : "HT6670300102",
"ANDROID_HOME" : "/home/vagrant/lisa/tools/android-sdk-linux/",
"exclude_modules" : [ "hwmon" ],
# List of additional devlib modules to install
"modules" : ['cgroups', 'bl', 'cpufreq'],
# List of additional binary tools to install
"tools" : ['rt-app', 'trace-cmd'],
# FTrace events to collect
"ftrace" : {
"events" : [
"sched_switch"
],
"buffsize" : 10240
}
}
te = TestEnv(my_conf, force_new=True)
target = te.target
# Report target connection
logging.info('Connected to %s target', target.abi)
print "DONE"
Details on the available controllers (or subsystems) can be found at: https://www.kernel.org/doc/Documentation/cgroup-v1/.
In [4]:
logging.info('%14s - Available controllers:', 'CGroup')
ssys = target.cgroups.list_subsystems()
for (n,h,g,e) in ssys:
logging.info('%14s - %10s (hierarchy id: %d) has %d cgroups',
'CGroup', n, h, g)
Cpusets provide a mechanism for assigning a set of CPUs and memory nodes to a set of tasks. Cpusets constrain the CPU and memory placement of tasks to only the resources available within a task's current cpuset. They form a nested hierarchy visible in a virtual file system. These are the essential hooks, beyond what is already present, required to manage dynamic job placement on large systems.
More information can be found in the kernel documentation: https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt.
In [5]:
# Get a reference to the CPUSet controller
cpuset = target.cgroups.controller('cpuset')
In [6]:
# Get the list of current configured CGroups for that controller
cgroups = cpuset.list_all()
logging.info('Existing CGropups:')
for cg in cgroups:
logging.info(' %s', cg)
In [7]:
# Dump the configuraiton of each controller
for cgname in cgroups:
#print cgname
cgroup = cpuset.cgroup(cgname)
attrs = cgroup.get()
#print attrs
cpus = attrs['cpus']
logging.info('%s:%-15s cpus: %s', cpuset.kind, cgroup.name, cpus)
In [8]:
# Create a LITTLE partition
cpuset_littles = cpuset.cgroup('/LITTLE')
In [9]:
# Check the attributes available for this control group
print "LITTLE:\n", json.dumps(cpuset_littles.get(), indent=4)
In [10]:
# Tune CPUs and MEMs attributes
# they must be initialize for the group to be usable
cpuset_littles.set(cpus=target.bl.littles, mems=0)
print "LITTLE:\n", json.dumps(cpuset_littles.get(), indent=4)
In [11]:
# Define a periodic big (80%) task
task = Periodic(
period_ms=100,
duty_cycle_pct=80,
duration_s=5).get()
# Create one task per each CPU in the target
tasks={}
for tid in enumerate(target.core_names):
tasks['task{}'.format(tid[0])] = task
# Configure RTA to run all these tasks
rtapp = RTA(target, 'simple', calibration=te.calibration())
rtapp.conf(kind='profile', params=tasks, run_dir=target.working_directory);
In [12]:
# Test execution of all these tasks into the LITTLE cluster
trace = rtapp.run(ftrace=te.ftrace, cgroup=cpuset_littles.name, out_dir=te.res_dir)
In [13]:
# Check tasks residency on little clsuter
trappy.plotter.plot_trace(trace)
In [14]:
# Compute and visualize tasks residencies on LITTLE clusterh CPUs
s = SchedMultiAssert(trappy.FTrace(trace), te.topology, execnames=tasks.keys())
residencies = s.getResidency('cluster', target.bl.littles, percent=True)
print json.dumps(residencies, indent=4)
In [15]:
# Assert that ALL tasks have always executed only on LITTLE cluster
s.assertResidency('cluster', target.bl.littles,
99.9, operator.ge, percent=True, rank=len(residencies))
Out[15]:
In [16]:
# Get a reference to the CPU controller
cpu = target.cgroups.controller('cpu')
In [17]:
# Create a big partition on that CPUS
cpu_littles = cpu.cgroup('/LITTLE')
In [18]:
# Check the attributes available for this control group
print "LITTLE:\n", json.dumps(cpu_littles.get(), indent=4)
In [19]:
# Set a 1CPU equivalent bandwidth for that CGroup
# cpu_littles.set(cfs_period_us=100000, cfs_quota_us=50000)
cpu_littles.set(shares=512)
print "LITTLE:\n", json.dumps(cpu_littles.get(), indent=4)
In [20]:
# Test execution of all these tasks into the LITTLE cluster
trace = rtapp.run(ftrace=te.ftrace, cgroup=cpu_littles.name)
In [21]:
# Check tasks residency on little cluster
trappy.plotter.plot_trace(trace)
In [22]:
# Isolate CPU0
# This works by moving all user-space tasks into a cpuset
# which does not include the specified list of CPUs to be
# isolated.
sandbox, isolated = target.cgroups.isolate(cpus=[0])
In [23]:
# Check the attributes available for the SANDBOX group
print "Sandbox:\n", json.dumps(sandbox.get(), indent=4)
In [24]:
# Check the attributes available for the ISOLATED group
print "Isolated:\n", json.dumps(isolated.get(), indent=4)
In [25]:
# Run some workload, which is expected to not run in the ISOLATED cpus:
trace = rtapp.run(ftrace=te.ftrace)
In [26]:
# Check tasks was not running on ISOLATED CPUs
trappy.plotter.plot_trace(trace)
In [27]:
# Compute and visualize tasks residencies on ISOLATED CPUs
s = SchedMultiAssert(trappy.FTrace(trace), te.topology, execnames=tasks.keys())
residencies = s.getResidency('cpu', [0], percent=True)
print json.dumps(residencies, indent=4)
In [28]:
# Assert that ISOLATED CPUs was not running workload tasks
s.assertResidency('cpu', [0], 0.0, operator.eq, percent=True, rank=len(residencies))
Out[28]: