In [1]:
import pandas as pd
import numpy as np
import seaborn as sb
import matplotlib.pyplot as plt
%matplotlib inline
This library is meant to provide tools for interactively exploring nanopore data inside the Jupyter notebook, and for writing simple scripts or more complex software dealing with nanopore data. Therefore a lot of attention has been given to make interactive use easy and painless, and to keep the code in the background flexible and exposed to library users.
The MinION sequencer is attached to a laptop running MinKnow. This program connects directly to the MinION device and tells it what to do. Optionally, third party software can connect to an API inside the primary software to remote control the sequencing process. That is not covered here, though.
In a nutshell, nanopore sequencing works by dragging a DNA molecule through a tiny pore in a membrane. As the DNA passes, the voltage difference between the two sides of the membrane change, depending on the electrochemical properties of the passing nucleotides. This means that, at the core of the nanopore data, there is a timeseries of voltage measurements, which is called the "squiggle".
The process to convert the squiggle into a sequence of DNA letters is called base calling. The current MinKnow software uploads the squiggle to Metrichor servers, which perform the base calling, and send the result back to the user's computer.
The result of a sequencing run is a collection of FAST5 files, each containing data on one molecule of DNA which passed through one of currently 512 channels in the flowcell. These files are stored on disk, usually in one directory per run. A convention seems to be to name each file with a unique and descriptive string:
In [2]:
!ls /home/andi/nanopore/GenomeRU2/downloads/pass/ | tail -n 10
These files belong to data publishd by Quick et al. http://europepmc.org/abstract/MED/25386338;jsessionid=ijHIHUVXlcpxeTzVUihz.0
In [3]:
import porekit
everything = porekit.gather_metadata("/home/andi/nanopore/", workers=4)
The result is a Pandas DataFrame object, which is too big to comfortably view in its entirety, but still comparatively "small data". Here is a subset of it:
In [ ]:
everything[['asic_id', 'channel_number', 'template_length', 'complement_length']].head()
All of the columns available:
In [ ]:
everything.columns
The philosopy of porekit is to gather the metadata once and then store it in a different format. This makes it easier and faster to analyse the metadata or use it in another context, for example with alignment data. In general, Fast5 Files don't change after MinKNOW and Metrichor, and possibly some third party programs are done with it.
The recommended file format to save this metadata is the "Feather File Format" (https://github.com/wesm/feather). It is a binary format for DataFrames, and has excellent support for R.
Feather's documentation emphasizes the file format can still change, and thus should not be used for long term storage. It's still fine for most medium term purposes, and even in the longer term, pinning the package versions will make older data available just as easily.
In [ ]:
everything.to_hdf("everything.h5", "meta")
In [ ]:
g = everything.groupby(['device_id', 'asic_id', 'run_id'])
In [ ]:
df = g.template_length.agg([lambda v: len(v), np.mean, np.max])
df.columns = ['Count', 'Mean template length', 'Max template_length']
df
As you can see, I have downloaded several nanopore sets from ENA. These are mostly incomplete sets, since I was interested more in the variety of data rather than the completeness. You can easily use wget to download a tarball from ENA, then extract the partial download. The last file will be truncated, but the rest is usable.