This notebook outlines input file types and their role in the FIDDLE work flow. It also outlines several ways of examining file contents and the variables lying within them that can be modified to suit a particular question. This notebook assumes that the quick start has been attempted by the user.
NOTE: The following python packages are easily installable via pip, e.g:
pip install h5py==2.7.0
In [1]:
from scipy import stats
import h5py
INTRO:
The highly modular nature of FIDDLE, exemplified by the depiction below, entails similarly modular input files.
RELEVANT JSON FILES:
There are two .json files that dictate how FIDDLE is run:
1. configurations.json
2. architecture.jsonconfigurations.json
'configurations.json' parametrizes the sequencing file input types and their characteristics. In the example case, the Genome sub-field is "sacCer3", the Tracks sub-field consist of TSS-seq data and others, the Options sub-field consist of which "Inputs", "Outputs", and other traits FIDDLE takes into consideration. Note that the caveat to the hyper-modularity of this input file is that each of the modified variables must exactly mirror what lies within the input hdf5 files - more on that down the page.
! cat configurations.json
In [ ]:
! cat configurations.json
architecture.json
'architecture.json' parametrizes the hyper-parameters and other neural network specific variables that FIDDLE will employ. The Encoder and Decoder will utilize the same hyper-parameters.
! cat architecture.json
In [ ]:
! cat architecture.json
RELEVANT HDF5 FILES:
Using the quick start hdf5 datasets as examples, one can see that the dimensions of the tracks within the hdf5 datasets reflect the characteristics of the sequencing inputs. The train, validation, and test hdf5datasets are simply partitions of an original hdf5dataset that was compiled from scripts found in the 'fiddle/data_prep/' directory. A guide on how this is carried out can be found by starting up the 'fiddle/data_prep/data_guide.ipynb'.
train = h5py.File('../data/hdf5datasets/train.h5', 'r')
train.items()
In [10]:
train = h5py.File('../data/hdf5datasets/NSMSDSRSCSTSRI_500bp/train.h5', 'r')
train.items()
Out[10]:
validation = h5py.File('../data/hdf5datasets/validation.h5', 'r')
validation.items()
In [8]:
validation = h5py.File('../data/hdf5datasets/NSMSDSRSCSTSRI_500bp/validation.h5', 'r')
validation.items()
Out[8]:
test = h5py.File('../data/hdf5datasets/test.h5', 'r')
test.items()
In [4]:
test = h5py.File('../data/hdf5datasets/NSMSDSRSCSTSRI_500bp/test.h5', 'r')
test.items()
Out[4]:
Examining the 'info' track:
The 'info' track is the track that holds index information relevant to the sequencing datasets. The dimensions of the 'info' correspond to the following:
1. Chromosome number (e.g. 1-16)
2. Strandedness (e.g. -1, 1)
3. Gene index (parsed from the original GFF file input)
4. Base Pair index (e.g. up to ~10^6)
infoRef_test = test.get('info')[:]
stats.describe(infoRef_test[:, X])
In [13]:
infoRef_test = test.get('info')[:]
stats.describe(infoRef_test[:, 0])
stats.describe(infoRef_test[:, 1])
stats.describe(infoRef_test[:, 2])
stats.describe(infoRef_test[:, 3])
print()
infoRef_validation = validation.get('info')[:]
stats.describe(infoRef_validation[:, 0])
stats.describe(infoRef_validation[:, 1])
stats.describe(infoRef_validation[:, 2])
stats.describe(infoRef_validation[:, 3])
print()
infoRef_train = train.get('info')[:]
stats.describe(infoRef_train[:, 0])
stats.describe(infoRef_train[:, 1])
stats.describe(infoRef_train[:, 2])
stats.describe(infoRef_train[:, 3])
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Jupyter Notebook as a Documentation Resource:
An advantage of this medium of Python interaction is the ability to quickly examine Python scripts. Using the '?' as a prepend, FIDDLE's documentation is easily accessed. First, we will import several FIDDLE scripts and quickly checkout their internals.
import main, models, analysis
In [ ]:
import main, models, visualization
Note: the following applies to any Python method/class/ADT/etc. as well as all imported Python packages:
The '?' prepend allows direct access to a Python script's docstrings. The '??' prepend allows direct access to the whole Python script. Jupyter Notebook's autocomplete feature allows an easy understanding of available methods. Click
?main
??models
?visualization
In [ ]:
?main
In [ ]:
??models.
In [ ]:
?visualization