Filtering and Ranking Security Bug Reports

This application is used to build prediction models for predicting security bug reports. Included here are bug reports from five different projects, code, experiments, and results. This document provides guidance on how to use the application on the command line. We start with some information about the bug reports and then we procede as follows:

• the generation of data matrices for experiments;
• calculation of tfidf as a measure of the popularity of security related keywords;
• prediction bug reports as security or non security reports and;
• ranking bug reports.

Users can clone the farsec repo and run the examples in order, using the commands in bold. The user must also create an output directory in the same location as farsec-v1.jar. All data and results will be sent to the output directory.

About Bug Reports

Bug Report Sources

• Chromium
• Wicket
• Ambari
• Camel
• Derby

Characteristics About Bug Reports

Bug Reports

For each project, data is partitioned into train and test sets. These are further partitioned into sbrs and nsbrs. Each bug report is converted into a file and stored in the approporiate directory (e.g -sbr-new, -nsbr-new, -sbr-old and -nsbr-old). An example of the result is shown in /resources/data1/wicket/

Data Matrices

Generate Data Matrices for Experiments

Given bug reports for each project we find security related keywords. We use the keywords to generate train and test sets with the bug reports. We generate these matrices for two types of prediction experiments. The first is within predition (WPP) the second is transfer prediction (TPP).

Example: WPP for wicket

Options:

• -o output directory name (mkdir)

• -p project name

• -n number of keywords/features

Run:

• java -jar farsec-v1.jar -o wicket-data1 -k data1 -p wicket -n 100 --wpp

Example: TPP for wicket (target) with ambari (source)

Options:

• -o output directory name (mkdir)
• -c source project name
• -p project name
• -n number of keywords/features

Run:

• java -jar farsec-v1.jar -o wicket-data1 -k data1 -c ambari -p wicket -n 100 --tpp

Generate Data Matrices for Experiments with Filtering

Filter WPP and TPP train data sets. These are denoted as WPPx and TPPx respectively.

Example: WPPx for wicket

Options:

• -o output directory name (mkdir)
• -c source project name
• -p project name
• -n number of keywords/features

Run:

• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket --wppx
• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket --tppx

Calculate Tf-idf

Generate tfidf Files

We use tf-idf as a proxy for popularity of specific keywords present is different sources. These sources are security bug reports and non security bug reports before and after filtering. The result is saved in a csv file.

Example for wicket

Options:

• -o output directory name (mkdir)

• -p project name

• -n number of keywords/features

Run:

• java -jar farsec-v1.jar -o wicket-data1 -p wicket -n 100 --tfidf

Make Predictions

We build prediction models using the following machine learning algorithms:

• Logistic Regression
• Naive Bayes
• Random Forest
• K-Nearest Neighbor
• Multilayer Perceptron

Bug reports in the test set are predicted as security or non security.

Options:

• -o output directory name (mkdir)
• -c source project name
• -p project name
• -n number of keywords/features

Run:

• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket -s wpp --predict
• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket -s wppx --predict
• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket -s tpp --predict
• java -jar farsec-v1.jar -o wicket-data1 -c ambari -p wicket -s tppx --predict