This notebook demonstrates how to evaluate classification accuracy of "cross-validated" simulated communities. Due to the unique nature of this analysis, the metrics that we use to evaluate classification accuracy are different from those used for mock.
The key measure here is rate of match vs. overclassification, hence P/R/F are not useful metrics. Instead, we define and measure the following as percentages:
Where L = taxonomic level being tested
In [8]:
from tax_credit.framework_functions import (novel_taxa_classification_evaluation,
extract_per_level_accuracy)
from tax_credit.eval_framework import parameter_comparisons
from tax_credit.plotting_functions import (pointplot_from_data_frame,
heatmap_from_data_frame,
per_level_kruskal_wallis,
rank_optimized_method_performance_by_dataset)
import seaborn.xkcd_rgb as colors
import pandas as pd
from os.path import expandvars, join, exists
from glob import glob
from IPython.display import display, Markdown
In [9]:
project_dir = "../.."
analysis_name = "cross-validated"
precomputed_results_dir = join(project_dir, "data", "precomputed-results", analysis_name)
expected_results_dir = join(project_dir, "data", analysis_name)
summary_fp = join(precomputed_results_dir, 'evaluate_classification_summary.csv')
results_dirs = glob(join(precomputed_results_dir, '*', '*', '*', '*'))
This cell performs the classification evaluation and should not be modified.
In [10]:
force = False
if force or not exists(summary_fp):
accuracy_results = novel_taxa_classification_evaluation(results_dirs, expected_results_dir,
summary_fp, test_type='cross-validated')
else:
accuracy_results = pd.DataFrame.from_csv(summary_fp)
Finally, we plot our results. Line plots show the mean +/- 95% confidence interval for each classification result at each taxonomic level (1 = phylum, 6 = species) in each dataset tested. Do not modify the cell below, except to adjust the color_pallette used for plotting. This palette can be a dictionary of colors for each group, as shown below, or a seaborn color palette.
match_ratio = proportion of correct matches.
underclassification_ratio = proportion of assignments to correct lineage but to a lower level than expected.
misclassification_ratio = proportion of assignments to an incorrect lineage.
In [11]:
color_palette={
'expected': 'black', 'rdp': colors['baby shit green'], 'sortmerna': colors['macaroni and cheese'],
'uclust': 'coral', 'blast': 'indigo', 'blast+': colors['electric purple'], 'naive-bayes': 'dodgerblue',
'naive-bayes-bespoke': 'blue', 'vsearch': 'firebrick'
}
level_results = extract_per_level_accuracy(accuracy_results)
y_vars = ['Precision', 'Recall', 'F-measure']
In [12]:
pointplot_from_data_frame(level_results, "level", y_vars,
group_by="Dataset", color_by="Method",
color_palette=color_pallette)
In [13]:
from pandas import DataFrame, concat, to_numeric
In [14]:
nb_results = level_results[level_results['Method'] == 'naive-bayes']
nb_results = nb_results.reset_index(drop=True)
columns = ['Alpha', 'kmer', 'Confidence']
def decode_params(p):
p = p.split(':')
p[-2] = int(eval(p[-2])[0])
return p
params = DataFrame((decode_params(s) for s in nb_results['Parameters']), columns=columns)
keepers = ['Dataset', 'level', 'Method']
metrics = ['Precision', 'Recall', 'F-measure']
raw_param_results = concat([nb_results[keepers + metrics], params], axis=1)
raw_param_results = raw_param_results.apply(to_numeric, errors='ignore')
param_results = raw_param_results.groupby(keepers + columns, as_index=False).mean()
param_results.level = param_results.level.astype(int)
param_results.kmer = param_results.kmer.astype(int)
len(param_results)
Out[14]:
In [15]:
level_pallete = {n:'blue' for n in range(1,6)}
level_pallete[6] = 'orange'
pointplot_from_data_frame(param_results, "kmer", y_vars,
group_by="Dataset", color_by="level",
color_palette=level_pallete)
Out[15]:
In [17]:
result = per_level_kruskal_wallis(level_results, y_vars, group_by='Method',
dataset_col='Dataset', alpha=0.05,
pval_correction='fdr_bh')
result
Out[17]:
In [18]:
heatmap_from_data_frame(level_results, metric="Precision", rows=["Method", "Parameters"], cols=["Dataset", "level"])
In [19]:
heatmap_from_data_frame(level_results, metric="Recall", rows=["Method", "Parameters"], cols=["Dataset", "level"])
In [20]:
heatmap_from_data_frame(level_results, metric="F-measure", rows=["Method", "Parameters"], cols=["Dataset", "level"])
Rank parameters for each method to determine the best parameter configuration within each method. Count best values in each column indicate how many samples a given method achieved within one mean absolute deviation of the best result (which is why they may sum to more than the total number of samples).
In [21]:
for method in level_results['Method'].unique():
top_params = parameter_comparisons(level_results, method, metrics=y_vars,
sample_col='Dataset', method_col='Method',
dataset_col='Dataset')
display(Markdown('## {0}'.format(method)))
display(top_params[:10])
Now we rank the top-performing method/parameter combination for each method at genus and species levels. Methods are ranked by top F-measure, and the average value for each metric is shown (rather than count best as above). F-measure distributions are plotted for each method, and compared using paired t-tests with FDR-corrected P-values. This cell does not need to be altered, unless if you wish to change the metric used for sorting best methods and for plotting.
In [22]:
rank_optimized_method_performance_by_dataset(level_results,
metric="F-measure",
level="level",
level_range=range(6,7),
display_fields=["Method",
"Parameters",
"Precision",
"Recall",
"F-measure"],
paired=True,
parametric=True,
color=None,
color_pallette=color_pallette)
In [ ]: