The following notebooks describe the evaluation of taxonomy classifiers using "novel taxa" data sets. Novel-taxa analysis is a form of cross-validated taxonomic classification, wherein random unique sequences are sampled from the reference database as a test set; all sequences sharing taxonomic affiliation at a given taxonomic level are removed from the reference database (training set); and taxonomy is assigned to the query sequences at the given taxonomic level. Thus, this test interrogates the behavior of a taxonomy classifier when challenged with "novel" sequences that are not represented by close matches within the reference sequence database. Such an analysis is performed to assess the degree to which "overassignment" occurs for sequences that are not represented in a reference database.
At each level
L, the unique taxonomic clades are randomly sampled and used as
QUERY sequences. All sequences that match that taxonomic annotation at
L are excluded from
REF. Hence, species-level
QUERY assignment asks how accurate assignment is to an "unknown" species that is not represented in the
REF, though other species in the same genus are. Genus-level
QUERY assignment asks how accurate assignment is to an "unknown" genus that is not represented in the
REF, though other genera in the same family are, et cetera.
The steps involved in preparing and executing novel-taxa analysis are described in a series of notebooks:
1) Novel taxa dataset generation only needs to be performed once for a given reference database. Only run this notebook if you wish to make novel taxa datasets from a different reference database, or alter the parameters used to make the novel taxa datasets. The default included in Tax-Credit is Greengenes 13_8 release, amplified in silico with primers 515f and 806r, and trimmed to 250 nt from the 5' end.
2) Taxonomic classification of novel taxa sequences is performed using the datasets generated in step 1. This template currently describes classification using QIIME 1 classifiers and can be used as a template for classifiers that are called via command line interface. Python-based classifiers can be used following the example of q2-feature-classifier.
3) Classifier evaluation is performed based on taxonomic classifications generated by each classifier used in step 2.
The dataset generation notebook uses a few novel definitions. The following provides some explanation of the definitions used in that notebook.
source= original reference database sequences and taxonomy.
QUERY= 'novel' query sequences and taxonomies randomly drawn from
noveltaxa, used for taxonomy assignment.
L= taxonomic level being tested
branching= describes a taxon at level
Lthat "branches" into two or more lineages at
L + 1.
REFmust contain one or more taxa that share the same clade at level
L - 1.
Lactobacillaceae └── Lactobacillus │ ├── Lactobacillus brevis │ └── Lactobacillus sanfranciscensis ├── Pediococcus │ ├── Pediococcus damnosus │ └── Pediococcus claussenii └── Paralactobacillus └── Paralactobacillus selangorensis
In [ ]: