In this short blog post, I want to show you an idea where you take some very detailed datasets from a software project and transform it into a representation where management can reason about. For this,
This little picture summarizes the main ideas where the data comes from:
Let's get some insights!
A proxy for utilization, we've taken the code coverage of a web application during the live operation with the user. We've collected that data by attaching the JaCoCo agent to the Java process (for details on how to do that see my blog post "Visualizing Production Coverage with JaCoCo, Pandas and D3").
The result is a dataset with covered lines per class. We read that into a Pandas DataFrame.
In [1]:
import pandas as pd
coverage = pd.read_csv("datasets/jacoco_production_coverage_spring_petclinic.csv")
coverage.head()
Out[1]:
We need to calculate some additional variables for our analysis later on: The lines of code per class and the ratio of the covered lines of code to all lines of code per class.
In [2]:
coverage['lines'] = coverage.LINE_MISSED + coverage.LINE_COVERED
coverage['covered'] = coverage.LINE_COVERED / coverage.lines
coverage.head()
Out[2]:
Next, we create a unique identifier for connecting the coverage data to the number of changes per class. We choose the fully qualified name ("fqn") of a class as key for this, set this variable as an index and only take the columns that we need later.
In [3]:
coverage['fqn'] = coverage.PACKAGE + "." + coverage.CLASS
coverage_per_class = coverage.set_index('fqn')[['lines', 'covered']]
coverage_per_class.head()
Out[3]:
As a proxy for the investments we took into our software, we choose the number of changes per file. For this, we need a log of our version control system. With Git, a simple numstat log does the job. We can get a list for each change of a file including the added and deleted lines with the command
git log --no-merges --no-renames --numstat --pretty=format:""We kind of hacked the git log pretty format option, but it does the job. We read that dataset into a DataFrame as well, using the tabular as a separator and specifying some column names manually.
In [4]:
changes = pd.read_csv(
"datasets/git_log_numstat_spring_petclinic.log",
sep="\t",
names=['additions', 'deletions', 'path'])
changes.head()
Out[4]:
As with the coverage data, we need a unique identifier per class. We can create that based on the path of each file.
In [5]:
changes['fqn'] = changes.path.str.extract(
"/java/(.*)\.java",
expand=True)[0]
changes['fqn'] = changes.fqn.str.replace("/", ".")
changes['fqn'][0]
Out[5]:
Different to the coverage dataset, we have multiple entries per file (for each change of that file). We need to group the changes by the fqn of each file or class respectively. We just take the path column that holds the number of changes for each class and renames that column to changes. The result is a Series of the number of changes per file.
In [6]:
changes_per_file = changes.groupby('fqn').path.count()
changes_per_file.name = "changes"
changes_per_file.head()
Out[6]:
In [7]:
analysis = coverage_per_class.join(changes_per_file)
analysis.head()
Out[7]:
OK, now to the key idea of this analysis. We need to raise the very fine-granular data on class level to a more higher-level where we can reason about in a better way. For this, we derive higher-level perspectives based on the fqn of the classes. For technical components, we can do this in our case based on the package name because the last part of the package name contains the name of the technical component.
In [8]:
analysis['tech'] = analysis.index.str.split(".").str[-2]
analysis.head()
Out[8]:
We can now aggregate our fine-granular data to a higher-level representation by grouping the data accordingly to the technical components. For the agg function, we provide a dictionary with the aggregation methods for each column.
In [9]:
tech_insights = analysis.groupby('tech').agg({
"lines" : "sum",
"covered": "mean",
"changes" : "sum"
})
tech_insights.head()
Out[9]:
With the help of a (self-written) little visualization, we can now create a SWOT matrix from our data.
In [10]:
%matplotlib inline
from lib.ausi import portfolio
portfolio.plot_diagram(tech_insights, "changes", "covered", "lines");
Discussion
jdbc component very often (aka invested much money) but the component isn't used at all. A clear failure of investments (OK, in this case, it's an alternative implementation of the database access, but we could also delete this code completely without any side effects).web component, but this isn't used as much as it should be (in relation to the changes). Maybe we can find code in this component that isn't needed anymore.petclinic (a base component), service, jpa and model components in the upper left are used very often with low changes (aka investments). This is good!
In [11]:
analysis['domain'] = "Other"
domains = ["Owner", "Pet", "Visit", "Vet", "Specialty", "Clinic"]
for domain in domains:
analysis.loc[analysis.index.str.contains(domain), 'domain'] = domain
analysis.head()
Out[11]:
Like with the technical components, we group the business aspects accordingly but also translate the technical terms into non-technical ones.
In [12]:
domain_insights = analysis.groupby('domain').agg({
"lines" : "sum",
"covered": "mean",
"changes" : "sum"
})
domain_insights = domain_insights.rename(columns=
{"lines": "Size", "covered" : "Utilization", "changes" : "Investment"})
domain_insights.head()
Out[12]:
Again, we plot a SWOT matrix with the data in the DataFrame.
In [13]:
portfolio.plot_diagram(domain_insights, "Investment", "Utilization", "Size");
Discussion
Clinic in the upper left of the matrix were worthwhile.Pet and Owner related classes because albeit we've invested heavily in this parts, they aren't used so much.Visit and Vet components in the lower right, we should take a look at if those components are worthwhile further improvements or candidates for removal.Other component has good chances to be improved the right way in the near future by only developing the parts further that are really needed.OK, we've seen how we can take some very fine-granular dataset, combine them and bring them to a higher-level perspective. We've also visualized the results of our analysis with a SWOT matrix to quickly spot parts of our software system that are good or bad investments.
What do you think about this approach? Do you like the idea? Or is this complete nonsense? Please leave me a comment!