Recentlym I came across the talk talk from the Diabolia
Among all the great tips, the vmstat command line utility seems to deliver great insights when it comes to quick performance analysis of Linux-based systems.
E. g., with a vmstat 1 60, vmstat will deliver you basic statistics about the utilization of your system frictionless:
It looks like a good idea to have a Jupyter notebook in place that can parse and visualize the data that vmstat produces. This could be very handy for quick performance analysis.
First, we parse our way through the data. I executed some tasks with high CPU load, downloaded something from the web and and copyied a huge file. I've recorded the results into the text file vmstat_load.log, which looks like this:
Yes, it's really ugly kind of input data with kind of fixed size columns. Nevertheless, let's get this into a nicely formated pandas dataframe.
In [1]:
import pandas as pd
vmstat_raw = pd.read_csv("datasets/vmstat_load90.log", sep="\n", header=None, skiprows=1, names=["raw"])
vmstat_raw.head(2)
Out[1]:
For getting rid of the fixed sized data, we can apply a little trick:
raw columns (with the default whitespace character). This gives us an array with all the non-whitespace parts.pd.Series accordingly. This gives us a collection Series (aka a DataFrame).
In [2]:
vmstat_temp = vmstat_raw['raw'].str.split().apply(pd.Series)
vmstat_temp.head(2)
Out[2]:
We also need suitable names for the columns of our newly DataFrame. For this, we can simply use the first columns of the new DataFrame.
In [3]:
vmstat_temp.columns = vmstat_temp.iloc[0]
vmstat_temp = vmstat_temp.dropna().reset_index(drop=True)
vmstat_temp.head()
Out[3]:
OK, let's start to convert the nicely formatted data now to the right data types:
to_numeric helper function on this data.to_datetime helper function on the data.
In [4]:
vmstat = vmstat_temp.iloc[:,:-2].apply(pd.to_numeric)
vmstat['UTC'] = pd.to_datetime(vmstat_temp['UTC'] + " " + vmstat_temp[None])
vmstat_timed = vmstat.set_index('UTC')
vmstat_timed.head(2)
Out[4]:
Last, because we have a timeseries with one entry per second, we can set the timestamp colum as index. This makes further time-based processing much easier.
In [5]:
vmstat_timed = vmstat.set_index('UTC')
vmstat_timed.head(2)
Out[5]:
In [6]:
from ozapfdis.linux import vmstat
vmstat_timed = vmstat.read_logfile("datasets/vmstat_load90.log")
In [7]:
vmstat_timed[['r', 'b']].plot();
In [8]:
vmstat_timed[['swpd', 'free', 'buff', 'cache']].plot();
In [9]:
vmstat_timed[['si', 'so']].plot();
In [10]:
vmstat_timed[['bi', 'bo']].plot();
In [11]:
vmstat_timed[['in', 'cs']].plot();
These are percentages of total CPU time.
In [12]:
vmstat_timed[['us', 'sy', 'id', 'wa', 'st']].plot.area();