This notebook gives an example of how to build relatively simple data paths. It assumes that you understand the concepts presented in the example 2 notebook.
The examples require that you understand a little bit about the example catalog data model, which is based on the FaceBase project.
'dataset' : represents a unit of data usually a 'study' or 'collection''experiment' : a bioassay (typically RNA-seq or ChIP-seq assays)'replicate' : a record of a replicate (bio or technical) related to an experimentdataset <- experiment: A dataset may have one to many experiments. I.e., there
is a foreign key reference from experiment to dataset.experiment <- replicate: An experiment may have one to many replicates. I.e., there is a
foreign key reference from replicate to experiment.
In [1]:
# Import deriva modules
from deriva.core import ErmrestCatalog, get_credential
In [2]:
# Connect with the deriva catalog
protocol = 'https'
hostname = 'www.facebase.org'
catalog_number = 1
# If you need to authenticate, use Deriva Auth agent and get the credential
credential = get_credential(hostname)
catalog = ErmrestCatalog(protocol, hostname, catalog_number, credential)
In [3]:
# Get the path builder interface for this catalog
pb = catalog.getPathBuilder()
In [4]:
dataset = pb.isa.dataset
experiment = pb.isa.experiment
replicate = pb.isa.replicate
In [5]:
path = dataset.path
print(path.uri)
In [6]:
path.link(experiment).link(replicate)
print(path.uri)
In [7]:
path.context.name
Out[7]:
In [8]:
entities = path.entities()
len(entities)
Out[8]:
Let's say we wanted to fetch the entities for the dataset table rather than the current context which is the replicate table. We can do that by referencing the table as a property of the path object. Note that these are known as "table instances" rather than tables when used within a path expression. We will discuss table instances later in this notebook.
In [9]:
path.table_instances['dataset']
# or
path.dataset
Out[9]:
From that table instance we can fetch entities, add a filter specific to that table instance, or even link another table. Here we will get the dataset entities from the path.
In [10]:
entities = path.dataset.entities()
len(entities)
Out[10]:
Notice that we fetched fewer entities this time which is the number of dataset entities rather than the replicate entities that we previously fetched.
Building off of the path, a filter can be added. Like fetching entities, linking and filtering are performed relative to the current context. In this filter, the assay's attriburtes are referenced in the expression.
Currently, binary comparisons and logical operators are supported. Unary opertors have not yet been implemented. In binary comparisons, the left operand must be an attribute (column name) while the right operand must be a literal value.
In [11]:
path.filter(replicate.bioreplicate_number == 1)
print(path.uri)
In [12]:
entities = path.entities()
len(entities)
Out[12]:
So far we have discussed base tables. A base table is a representation of the table as it is stored in the ERMrest catalog. A table instance is a usage or reference of a table within the context of a data path. As demonstrated above, we may link together multiple tables and thus create multiple table instances within a data path.
For example, in path.link(dataset).link(experiment).link(replicate) the table instance experiment is no longer the same as the original base table experiment because within the context of this data path the experiment entities must satisfy the constraints of the data path. The experiment entities must reference a dataset entity, and they must be referenced by a replicate entity. Thus within this path, the entity set for experiment may be quite different than the entity set for the base table on its own.
Whenever you initiate a data path (e.g., table.path) or link a table to a path (e.g., path.link(table)) a table instance is created and bound to the DataPath object (e.g., path). These table instances can be referenced via the DataPath's table_instances container or directly as a property of the DataPath object itself.
In [13]:
dataset_instance = path.table_instances['dataset']
# or
dataset_instance = path.dataset
Whenever a table instance is created and bound to a path, it is given a name. If no name is specified for it, it will be named after the name of its base table. For example, a table named "My Table" will result in a table instance also named "My Table". Tables may appear more than once in a path (as table instances), and if the table name is taken, the instance will be given the "'base name' + number" (e.g., "My Table2").
You may wish to specify the name of your table instance. In conventional database terms, an alternate name is called an "alias". Here we give the dataset table instance an alias of 'D' though longer strings are also valid as long as they do not contain special characters in them.
In [14]:
path.link(dataset.alias('D'))
Out[14]:
In [15]:
path.D.uri
Out[15]:
You'll notice that in this path we added an additional instance of the dataset table from our catalog model. In addition, we linked it to the isa.replicate table. This was possible because in this model, there is a foriegn key reference from the base table replicate to the base table dataset. The entities for the table instance named dataset and the instance name D will likely consist of different entities because the constraints for each are different.
In [16]:
path = dataset.path.link(experiment).link(replicate).filter(replicate.bioreplicate_number == 1)
print(path.uri)
Now let's fetch an entity set with attributes pulled from each of the table instances in the path.
In [17]:
results = path.attributes(path.dataset.accession,
path.experiment.experiment_type.alias('type_of_experiment'),
path.replicate.technical_replicate_number.alias('technical_replicate_num'))
print(results.uri)
Notice that the ResultSet also has a uri property. This URI may differ from the origin path URI because the attribute projection does not get appended to the path URI.
In [18]:
path.uri != results.uri
Out[18]:
As usual, fetch(...) the entities from the catalog.
In [19]:
results.fetch(limit=5)
for result in results:
print(result)
In [ ]: