This IPython Notebook illustrates how to apply the evaluation framework described in (Bokulich, Rideout, et al. (in preparation)). Given a set of precomputed taxonomic assignment method evaluation results (this will likely be the ones included in the short-read-tax-assignment GitHub repository) and a set of results generated by the user for a new method, the results from the new method are analyzed in the context of the precomputed results. This allows users to rapidly determine how a new method performs, relative to the precomputed results.
The following sections present different evaluations with an example usearch-based taxonomic assigner. A parameter sweep was performed using this new taxonomic assigner (as described in the usearch-tax-assigner notebook), and the evaluations that are run here allow us to determine how the new assigner performs relative to the precomputed results and summarize the results in a report, which will be this notebook after all cells have been executed. This provides a convenient interactive framework for analyzing new methods.
This notebook could easily be applied to the results of a different taxonomic assigner, and it is therefore intended to serve as a template for report generation. Only one change would need to be made to support that, and that change is discussed inline below.
This code and other components of this framework can be found in the short-read-tax-assignment GitHub repository.
To run this notebook, you'll need to have the following installed:
Throughout the notebook and associated code, we refer to the precomputed results obtained from GitHub as the subject results, and the results generated by the user as the query results.
To prepare results from another classifier for analysis, you'll need to have BIOM files with taxonomy assignments as an observation metadata category called taxonomy. These should be generated for all analyses (mock community, natural community), datasets, reference databases, methods, and parameter combinations of interest. An example of how to generate these is presented in the usearch-tax-assigner notebook).
Your BIOM tables should be called table.biom, and nested in the following directory structure:
query_results_dir/
analysis/
dataset-id/
reference-db-id/
method-id/
parameter-combination-id/
table.biomquery_results_dir is the name of the top level directory, and you will set this value in the first code cell of this notebook. You can name this directory whatever you want to.
This directory structure is identical to that for the subject results. You can review that directory structure for an example of how this should look.
In [1]:
## short_read_tax_dir should be the directory where you've downloaded (or cloned) the
## short-read-tax-assignment repository.
short_read_tax_dir = "/home/jrideout/dev/short-read-tax-assignment/"
## query_results_dir should be the directory where the results of the method(s) to be
## compared to the pre-computed results can be found.
query_results_dir = "/home/jrideout/dev/short-read-tax-assignment/usearch_parameter_sweep/"
In [2]:
%matplotlib inline
from os.path import join, exists
from taxcompare.eval_framework import (get_expected_tables_lookup,
find_and_process_result_tables,
compute_prfs,
compute_pearson_spearman,
compute_procrustes,
generate_pr_scatter_plots,
generate_prf_box_plots,
generate_precision_box_plots,
generate_recall_box_plots,
generate_fmeasure_box_plots,
generate_prf_table,
generate_correlation_box_plots,
generate_pearson_box_plots,
generate_spearman_box_plots,
generate_pearson_spearman_table,
generate_procrustes_table)
from cogent.draw.distribution_plots import generate_box_plots
In [3]:
# Define the subdirectories where the query mock and natural community data should be, and confirm that they exist.
mock_query_results_dir = join(query_results_dir,"mock")
natural_query_results_dir = join(query_results_dir,"natural")
assert exists(mock_query_results_dir), "Mock community query directory doesn't exist: %s" % mock_query_results_dir
assert exists(natural_query_results_dir), "Natural community query directory doesn't exist: %s" % natural_query_results_dir
# Define the subdirectories where the subject mock and natural community data should be, and confirm that they exist.
mock_subject_results_dir = join(short_read_tax_dir,'data/eval-subject-results/mock/')
natural_subject_results_dir = join(short_read_tax_dir,'data/eval-subject-results/natural/')
assert exists(mock_query_results_dir), "Mock community subject directory doesn't exist: %s" % mock_subject_results_dir
assert exists(natural_query_results_dir), "Natural community subject directory doesn't exist: %s" % natural_subject_results_dir
Next we'll use the paths defined above to find all of the tables that will be compared. These include the pre-computed result tables (i.e., the ones that the new methods will be compared to), the expected result tables (i.e., the tables containing the known composition of the mock microbial communities), and the query result tables (i.e., the tables generated with the new method(s) that we want to compare to the pre-computed result tables).
In [4]:
query_results = find_and_process_result_tables(mock_query_results_dir)
subject_results = find_and_process_result_tables(mock_subject_results_dir)
expected_L6_tables = get_expected_tables_lookup(mock_subject_results_dir)
In this evaluation, we compute and summarize precision, recall, and F-measure of each result (pre-computed and query) based on the known composition of the mock communities. We then summarize the results in two ways: first with a scatter plot, where pre-computed results are represented by blue points, and query results are represented by red points; and second with a table of the top twenty-five methods based on their F-measures.
This is a qualitative evaluation, effectively telling us about the ability of the different methods to report the taxa that are present in each sample. These metrics are not concerned with the abundance of the different taxa.
In [5]:
subject_prf = list(compute_prfs(subject_results,expected_L6_tables))
In [6]:
query_prf = list(compute_prfs(query_results,expected_L6_tables))
In [7]:
generate_pr_scatter_plots(query_prf,subject_prf)
In [8]:
generate_precision_box_plots(query_prf,subject_prf)
In [9]:
generate_recall_box_plots(query_prf,subject_prf)
In [10]:
generate_fmeasure_box_plots(query_prf,subject_prf)
In [11]:
generate_prf_table(query_prf,subject_prf)
In this evaluation, we compute and summarize the correlation between each result (pre-computed and query) and the known composition of the mock communities. We then summarize the results in two ways: first with a series of boxplots of correlation coefficients by method; and second with a table of the top twenty-five methods based on their Pearson correlation coefficient.
This is a quantitative evaluation, which tells us about the ability of the different methods to report the taxa that are present in each sample and accurately assess their abundance. Because many factors can affect the observed abundance of taxa beyond the accuracy of the taxonomic assigner (e.g., primer bias), the correlation coefficients are frequently low, but we expect that their relative values are informative in understanding which taxonomic assigners are more correct than others.
In [12]:
subject_pearson_spearman = list(compute_pearson_spearman(subject_results,expected_L6_tables))
In [13]:
query_pearson_spearman = list(compute_pearson_spearman(query_results,expected_L6_tables))
In [14]:
generate_pearson_box_plots(query_pearson_spearman,subject_pearson_spearman)
In [15]:
generate_spearman_box_plots(query_pearson_spearman,subject_pearson_spearman)
In [16]:
generate_pearson_spearman_table(query_pearson_spearman,subject_pearson_spearman)
Next we'll use the paths defined above to find all of the natural community tables that will be compared. These include the pre-computed result tables (i.e., the ones that the new methods will be compared to), the expected result tables (i.e., the OTU table), and the query result tables (i.e., the tables generated with the new method(s) that we want to compare to the pre-computed result tables).
In [17]:
query_results = find_and_process_result_tables(natural_query_results_dir,filename_pattern="table.biom")
subject_results = find_and_process_result_tables(natural_subject_results_dir)
expected_pcs = get_expected_tables_lookup(natural_subject_results_dir,
filename_pattern='bray_curtis_pc.txt')
In [18]:
query_procrustes = list(compute_procrustes(query_results,expected_pcs))
In [19]:
subject_procrustes = list(compute_procrustes(subject_results,expected_pcs))
In [20]:
generate_procrustes_table(query_procrustes,subject_procrustes)
In [ ]: