Notes:
Questions:
Do we want an Identifier Type separate from Entity and Relation identifiers? I think we do, so we can specify the entity type(s) a given identifier should be used on.
In [1]:
me = "filter-network-relations-dev"
In [2]:
debug_flag = False
In [3]:
import datetime
from django.db.models import Avg, Max, Min, Q
from django.utils.text import slugify
import json
import logging
import six
In [4]:
%pwd
Out[4]:
In [5]:
# current working folder
current_working_folder = "/home/jonathanmorgan/work/django/research/work/phd_work/analysis"
current_datetime = datetime.datetime.now()
current_date_string = current_datetime.strftime( "%Y-%m-%d-%H-%M-%S" )
configure logging for this notebook's kernel (If you do not run this cell, you'll get the django application's logging configuration.
In [6]:
logging_file_name = "{}/logs/{}-{}.log.txt".format( current_working_folder, me, current_date_string )
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
filename = logging_file_name,
filemode = 'w' # set to 'a' if you want to append, rather than overwrite each time.
)
print( "Logging initialized, to {}".format( logging_file_name ) )
If you are using a virtualenv, make sure that you:
Since I use a virtualenv, need to get that activated somehow inside this notebook. One option is to run ../dev/wsgi.py in this notebook, to configure the python environment manually as if you had activated the sourcenet virtualenv. To do this, you'd make a code cell that contains:
%run ../dev/wsgi.py
This is sketchy, however, because of the changes it makes to your Python environment within the context of whatever your current kernel is. I'd worry about collisions with the actual Python 3 kernel. Better, one can install their virtualenv as a separate kernel. Steps:
activate your virtualenv:
workon researchin your virtualenv, install the package ipykernel.
pip install ipykerneluse the ipykernel python program to install the current environment as a kernel:
python -m ipykernel install --user --name <env_name> --display-name "<display_name>"
sourcenet example:
python -m ipykernel install --user --name sourcenet --display-name "research (Python 3)"More details: http://ipython.readthedocs.io/en/stable/install/kernel_install.html
First, initialize my dev django project, so I can run code in this notebook that references my django models and can talk to the database using my project's settings.
In [7]:
# init django
django_init_folder = "/home/jonathanmorgan/work/django/research/work/phd_work"
django_init_path = "django_init.py"
if( ( django_init_folder is not None ) and ( django_init_folder != "" ) ):
# add folder to front of path.
django_init_path = "{}/{}".format( django_init_folder, django_init_path )
#-- END check to see if django_init folder. --#
In [8]:
%run $django_init_path
In [9]:
# context imports
from context.export.network.filter_spec import FilterSpec
from context.export.network.network_data_request import NetworkDataRequest
from context.models import Entity
from context.models import Entity_Identifier_Type
from context.models import Entity_Identifier
from context.models import Entity_Relation
from context.models import Entity_Type
from context.tests.export.network.test_helper import TestHelper
# context_text imports
from context_text.article_coding.article_coding import ArticleCoder
from context_text.article_coding.article_coding import ArticleCoding
from context_text.article_coding.open_calais_v2.open_calais_v2_article_coder import OpenCalaisV2ArticleCoder
from context_text.collectors.newsbank.newspapers.GRPB import GRPB
from context_text.collectors.newsbank.newspapers.DTNB import DTNB
from context_text.export.to_context_base.export_to_context import ExportToContext
from context_text.models import Article
from context_text.models import Article_Subject
from context_text.models import Newspaper
from context_text.shared.context_text_base import ContextTextBase
Create a LoggingHelper instance to use to log debug and also print at the same time.
Preconditions: Must be run after Django is initialized, since python_utilities is in the django path.
In [11]:
# python_utilities
from python_utilities.logging.logging_helper import LoggingHelper
# init
my_logging_helper = LoggingHelper()
my_logging_helper.set_logger_name( me )
log_message = None
Now, we need to render out our network data from context, so we can then test it out and make sure we are getting the same answers we got from the old way.
First step is to translate the filter criteria for nodes and ties from the existing admin for the querying context.
Configuration of Network Builder, from methods-network_analysis-create_network_data.ipynb:
Configuration to generate network files for prelim:
Config of "Select Articles" - fields in bold need to be changed from default values:
Start date (YYYY-MM-DD): 2009-12-01End date (YYYY-MM-DD): 2009-12-31Fancy date range: - Empty.Publications: "Grand Rapids Press, The"Coders: None selected.Coder IDs to include, in order of highest to lowest priority:
if automated: Article_Data coder_type Filter Type and coder_type 'Value In' List (comma-delimited):
use the coder_type filter fields to filter automatically coded Article_Data on coder type if you have tried different automated coder types:
Article_Data coder_type Filter Type: - Just automatedcoder_type 'Value In' List (comma-delimited): - Enter the coder types you want included. Examples:
Topics: None selected.
Article Tag List (comma-delimited): - "grp_month"Unique Identifier List (comma-delimited): - Empty.Allow duplicate articles: - "No"Configure "Network Settings" - fields in bold need to be changed from default values:
relations - Include source contact types - All selected.relations - Include source capacities: - None selected.relations - Exclude source capacities: - None selected.Download as File? - "Yes"Include render details? - "No"Data Format: - "Tab-Delimited Matrix"Data Output Type: - "Network + Attribute Columns"Network Label: - Empty.Include Headers: - "Yes"Config of "Select People" - fields in bold need to be changed from default values:
Person Query Type: - "Custom, defined below"People from (YYYY-MM-DD): - 2009-12-01People to (YYYY-MM-DD): - 2009-12-31Fancy person date range: - Empty.Person publications: - "Grand Rapids Press, The"Person coders: - "automated", "minnesota1", "minnesota2", "minnesota3", "ground_truth"Coder IDs to include, in order of highest to lowest priority: - Empty.Article_Data coder_type Filter Type and coder_type 'Value In' List (comma-delimited):
use the coder_type filter fields to filter automatically coded Article_Data on coder type if you have tried different automated coder types:
Article_Data coder_type Filter Type: - Just automatedcoder_type 'Value In' List (comma-delimited): - Enter the coder types you want included. Examples:
Person Topics: None
Article Tag List (comma-delimited): - "grp_month"Unique Identifier List (comma-delimited): - Empty.Person allow duplicate articles: - "Yes"Below is a JSON file that is just the automated coding for the month from 2009-12-01 through 2009-12-31. Just about all of the complexity of the original screens is possible here, as long as you loaded the entities and ties and all of the needed traits, including some way of adding tags...
{
"output_specification": {
"output_type": "file",
"output_file_path": "./NetworkDataRequest_test_output.txt",
"output_format": "TSV_matrix",
"output_structure": "both_trait_columns",
"output_include_column_headers": true
},
"relation_selection": {
"relation_type_slug_filter_combine_type": "AND",
"relation_type_slug_filters": [
{
"comparison_type": "includes",
"value_list": [ "mentioned", "qouted", "shared_byline" ]
}
],
"relation_trait_filter_combine_type": "AND",
"relation_trait_filters": [
{
"name": "pub_date",
"data_type": "date",
"comparison_type": "in_range",
"value_from": "2009-12-01",
"value_to": "2009-12-31"
},
{
"name": "sourcenet-coder-User-username",
"data_type": "string",
"comparison_type": "includes",
"value_list": [ "automated" ]
},
{
"name": "coder_type",
"data_type": "string",
"comparison_type": "includes",
"value_list": [ "OpenCalais_REST_API_v2" ]
}
],
"entity_type_slug_filter_combine_type": "AND",
"entity_type_slug_filters": [
{
"comparison_type": "includes",
"value_list": [ "person" ],
"relation_roles_list": [ "FROM" ]
},
{
"comparison_type": "includes",
"value_list": [ "person" ],
"relation_roles_list": [ "TO" ]
},
{
"comparison_type": "includes",
"value_list": [ "article" ],
"relation_roles_list": [ "THROUGH" ]
}
],
"entity_trait_filter_combine_type": "AND",
"entity_trait_filters": [
{
"name": "sourcenet-Newspaper-ID",
"data_type": "int",
"comparison_type": "includes",
"value_list": [ 1 ],
"relation_roles_list": [ "THROUGH" ]
}
]
}
}To start, try making a fixture of all of the context app from the temp database. This should be entities and ties for the 50 or so articles we coded there.
This includes contents of file "context-sourcenet_entity_and_relation_types.json", so you can just load this one file.
Export to JSON fixture files using manage.py / django-admin dumpdata ( https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-dumpdata ) so they can be imported using python manage.py or django-admin loaddata ( https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-loaddata ) rather than having to input them in the admin:
python manage.py dumpdata [app_label[.ModelName] [app_label[.ModelName] ...]] --indent INDENT --output <output_file_path>
python manage.py dumpdata \
--indent 4 \
--output context-sourcenet_entities_and_relations-full.json \
context
No line breaks:
python manage.py dumpdata --indent 4 --output context-sourcenet_entities_and_relations-full.json contextThis excludes contents of file "context-sourcenet_entity_and_relation_types.json", so load that first, then this.
Export to JSON fixture files using manage.py / django-admin dumpdata ( https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-dumpdata ) so they can be imported using python manage.py or django-admin loaddata ( https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-loaddata ) rather than having to input them in the admin:
python manage.py dumpdata [app_label[.ModelName] [app_label[.ModelName] ...]] --indent INDENT --output <output_file_path>
python manage.py dumpdata \
--indent 4 \
--output context-sourcenet_entities_and_relations.json \
--exclude context.Entity_Identifier_Type \
--exclude context.Entity_Relation_Type \
--exclude context.Entity_Relation_Type_Trait \
--exclude context.Entity_Type \
--exclude context.Entity_Type_Trait \
--exclude context.Trait_Type \
--exclude context.Term \
--exclude context.Term_Relation \
--exclude context.Term_Relation_Type \
--exclude context.Vocabulary \
context
No line breaks:
python manage.py dumpdata --indent 4 --output context-sourcenet_entities_and_relations.json --exclude context.Entity_Identifier_Type --exclude context.Entity_Relation_Type --exclude context.Entity_Relation_Type_Trait --exclude context.Entity_Type --exclude context.Entity_Type_Trait --exclude context.Trait_Type --exclude context.Term --exclude context.Term_Relation --exclude context.Term_Relation_Type --exclude context.Vocabulary contextFor each type of filter, need to prototype the code to implement the filter.
Figure out which relation roles we are focusing on (combination of FROM, TO, THROUGH), then build out query to properly filter.
In [12]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
entity_qs = entity_qs.filter( entity_identifier__name = "person_sourcenet_id" )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
entity_qs = entity_qs.filter( entity_identifier__uuid = "202" )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [13]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
combined_q = Q( entity_identifier__name = "person_sourcenet_id" ) & Q( entity_identifier__uuid = "202" )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [14]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# try to find relations to an Entity in any relation role (FROM, TO, THROUGH), with identifier name "`person_sourcenet_id`" and id 202.
from_q = Q( relation_from__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_from__entity_identifier__uuid = "202" )
relation_qs = relation_qs.filter( from_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [15]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# try to find relations to an Entity in any relation role (FROM, TO, THROUGH), with identifier name "`person_sourcenet_id`" and id 202.
to_q = Q( relation_to__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_to__entity_identifier__uuid = "202" )
relation_qs = relation_qs.filter( to_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [16]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# try to find relations to an Entity in any relation role (FROM, TO, THROUGH), with identifier name "`person_sourcenet_id`" and id 202.
through_q = Q( relation_through__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_through__entity_identifier__uuid = "202" )
relation_qs = relation_qs.filter( through_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [17]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# try to find relations to an Entity in any relation role (FROM, TO, THROUGH), with identifier name "`person_sourcenet_id`" and id 202.
from_q = Q( relation_from__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_from__entity_identifier__uuid = "202" )
to_q = Q( relation_to__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_to__entity_identifier__uuid = "202" )
through_q = Q( relation_through__entity_identifier__name = "person_sourcenet_id" ) & Q( relation_through__entity_identifier__uuid = "202" )
combined_q = from_q | to_q | through_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [18]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
entity_qs = Entity.objects.filter( pk = 315 )
# try to find relations to an Entity in any relation role (FROM, TO, THROUGH), with identifier name "`person_sourcenet_id`" and id 202.
from_q = Q( relation_from__in = entity_qs )
combined_q = from_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
to_q = Q( relation_to__in = entity_qs )
combined_q = combined_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
through_q = Q( relation_through__in = entity_qs )
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [19]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "46" )
test_value_list.append( "163" )
test_value_list.append( "161" )
test_value_list.append( "164" )
test_value_list.append( "30" )
test_value_list.append( "175" )
combined_q = Q( entity_identifier__name = "person_sourcenet_id" ) & Q( entity_identifier__uuid__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {}".format( entity_id ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [20]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [21]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "20202020" )
test_value_list.append( "20202021" )
test_value_list.append( "20202022" )
id_name_q = Q( entity_identifier__name = "person_sourcenet_id" )
id_value_q = Q( entity_identifier__uuid__in = test_value_list )
combined_q = id_name_q & ( ~ id_value_q )
#entity_qs = entity_qs.filter( id_name_q )
#entity_qs = entity_qs.exclude( id_value_q )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [22]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [23]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "46" )
test_value_list.append( "163" )
test_value_list.append( "161" )
test_value_list.append( "164" )
test_value_list.append( "30" )
test_value_list.append( "175" )
id_name_q = Q( entity_identifier__name = "person_sourcenet_id" )
id_value_q = Q( entity_identifier__uuid__in = test_value_list )
combined_q = id_name_q & ( ~ id_value_q )
#entity_qs = entity_qs.filter( id_name_q )
#entity_qs = entity_qs.exclude( id_value_q )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [24]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
Figure out which relation roles we are focusing on (combination of FROM, TO, THROUGH), then build out query to properly filter.
In [25]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
entity_qs = entity_qs.filter( entity_trait__name = "first_name" )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
entity_qs = entity_qs.filter( entity_trait__value = "John" )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [26]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
combined_q = Q( entity_trait__name = "first_name" ) & Q( entity_trait__value = "Larry" )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [27]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [28]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "John" )
test_value_list.append( "Michael" )
test_value_list.append( "Robert" )
test_value_list.append( "Steve" )
test_value_list.append( "Larry" )
trait_name_q = Q( entity_trait__name = "first_name" )
trait_value_q = Q( entity_trait__value__in = test_value_list )
combined_q = trait_name_q & trait_value_q
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [29]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [30]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "20202020" )
test_value_list.append( "20202021" )
test_value_list.append( "20202022" )
trait_name_q = Q( entity_trait__name = "first_name" )
trait_value_q = ~ Q( entity_trait__value__in = test_value_list )
combined_q = trait_name_q & trait_value_q
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [31]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [32]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "John" )
test_value_list.append( "Michael" )
test_value_list.append( "Robert" )
test_value_list.append( "Steve" )
test_value_list.append( "Larry" )
trait_name_q = Q( entity_trait__name = "first_name" )
trait_value_q = ~ Q( entity_trait__value__in = test_value_list )
combined_q = trait_name_q & trait_value_q
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [33]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [34]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
trait_name_q = Q( entity_trait__name = "pub_date" )
left_range_q = Q( entity_trait__value__gte = "2010-02-08" )
right_range_q = Q( entity_trait__value__lte = "2010-02-13" )
combined_q = trait_name_q & left_range_q & right_range_q
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [35]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
Figure out which relation roles we are focusing on (combination of FROM, TO, THROUGH), then build out query to properly filter.
In [36]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
entity_qs = entity_qs.filter( entity_types__entity_type__slug = "person" )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [37]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
combined_q = Q( entity_types__entity_type__slug = "person" )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [38]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [39]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "person" )
test_value_list.append( "article" )
combined_q = Q( entity_types__entity_type__slug__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [40]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | THROUGH
combined_q = from_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO | THROUGH
combined_q = to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nTO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = from_q | to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [41]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "peregrine" )
test_value_list.append( "chartreuse" )
test_value_list.append( "bumblebee" )
combined_q = ~ Q( entity_types__entity_type__slug__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [42]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [43]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "peregrine" )
test_value_list.append( "chartreuse" )
test_value_list.append( "bumblebee" )
test_value_list.append( "article" )
combined_q = ~ Q( entity_types__entity_type__slug__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [44]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [45]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
name_q = Q( entity_relation_trait__name = "pub_date" )
value_q = Q( entity_relation_trait__value = "2009-12-07" )
combined_q = name_q & value_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [46]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "20202020" )
test_value_list.append( "20202021" )
test_value_list.append( "20202022" )
name_q = Q( entity_relation_trait__name = "pub_date" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
combined_q = name_q & value_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [47]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "2009-12-07" )
test_value_list.append( "2010-02-08" )
test_value_list.append( "2010-02-13" )
name_q = Q( entity_relation_trait__name = "pub_date" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
combined_q = name_q & value_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [48]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "20202020" )
test_value_list.append( "20202021" )
test_value_list.append( "20202022" )
name_q = Q( entity_relation_trait__name = "pub_date" )
value_q = ~ Q( entity_relation_trait__value__in = test_value_list )
combined_q = name_q & value_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [49]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "2009-12-07" )
test_value_list.append( "2010-02-08" )
test_value_list.append( "2010-02-13" )
name_q = Q( entity_relation_trait__name = "pub_date" )
value_q = ~ Q( entity_relation_trait__value__in = test_value_list )
combined_q = name_q & value_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [50]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
name_q = Q( entity_relation_trait__name = "pub_date" )
left_range_q = Q( entity_relation_trait__value__gte = "2019-12-01" )
right_range_q = Q( entity_relation_trait__value__lte = "2019-12-31" )
combined_q = name_q & left_range_q & right_range_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [51]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
name_q = Q( entity_relation_trait__name = "pub_date" )
left_range_q = Q( entity_relation_trait__value__gte = "2010-02-08" )
right_range_q = Q( entity_relation_trait__value__lte = "2010-02-13" )
combined_q = name_q & left_range_q & right_range_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [52]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
name_q = Q( relation_type__slug = "quoted" )
combined_q = name_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [53]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "quoted" )
test_value_list.append( "mentioned" )
test_value_list.append( "shared_byline" )
name_q = Q( relation_type__slug__in = test_value_list )
combined_q = name_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [54]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "quoted" )
test_value_list.append( "mentioned" )
test_value_list.append( "shared_byline" )
name_q = ~ Q( relation_type__slug__in = test_value_list )
combined_q = name_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
In [55]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "person" )
test_value_list.append( "article" )
combined_q = Q( entity_types__entity_type__slug__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [56]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | THROUGH
combined_q = from_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO | THROUGH
combined_q = to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nTO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = from_q | to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [57]:
# try to just find the entity with the desired identifiers.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
test_value_list = []
test_value_list.append( "newspaper" )
test_value_list.append( "article" )
combined_q = Q( entity_types__entity_type__slug__in = test_value_list )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# loop to retrieve IDs.
entity_id_list = []
for entity in entity_qs:
# get, print, and store ID.
entity_id = entity.id
print( "Entity ID: {} ( {} )".format( entity_id, entity.name ) )
entity_id_list.append( entity_id )
#-- END loop over matching entities. --#
In [58]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | THROUGH
combined_q = from_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO | THROUGH
combined_q = to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nTO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = from_q | to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [59]:
# start with all Entity_Relations.
#test_relation_qs = None
test_relation_qs = Entity_Relation.objects.all()
In [60]:
# try to just find the entity with the desired identifiers.
# two QuerySets, one for just authors, and one for just articles.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# Q for each of article and person entities.
article_entity_q = Q( entity_types__entity_type__slug = "article" )
person_entity_q = Q( entity_types__entity_type__slug = "person" )
# Make a QuerySet for each.
article_entity_qs = entity_qs.filter( article_entity_q )
print( "\narticle entity result count: {} (SQL: {})".format( article_entity_qs.count(), article_entity_qs.query ) )
person_entity_qs = entity_qs.filter( person_entity_q )
print( "\nperson entity result count: {} (SQL: {})".format( person_entity_qs.count(), person_entity_qs.query ) )
In [61]:
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = person_entity_qs )
to_q = Q( relation_to__in = person_entity_qs )
through_q = Q( relation_through__in = article_entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | THROUGH
combined_q = from_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO | THROUGH
combined_q = to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nTO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = from_q | to_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM & TO & THROUGH
combined_q = from_q & to_q & through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM & TO & THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM & TO & THROUGH - filter
test_qs = relation_qs.filter( from_q )
test_qs = test_qs.filter( to_q )
test_qs = test_qs.filter( through_q )
print( "\nFROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( from_q )
test_relation_qs = test_relation_qs.filter( to_q )
test_relation_qs = test_relation_qs.filter( through_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [62]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
name_q = Q( entity_relation_trait__name = "pub_date" )
left_range_q = Q( entity_relation_trait__value__gte = "2009-12-01" )
right_range_q = Q( entity_relation_trait__value__lte = "2009-12-31" )
pub_date_q = name_q & left_range_q & right_range_q
test_qs = relation_qs.filter( pub_date_q )
print( "\npub_date result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( pub_date_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [63]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
test_value_list = []
test_value_list.append( "automated" )
name_q = Q( entity_relation_trait__name = "sourcenet-coder-User-username" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
coder_username_q = name_q & value_q
relation_qs = relation_qs.filter( coder_username_q )
print( "\ncoder_username result count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( coder_username_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [64]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
test_value_list = []
test_value_list.append( "OpenCalais_REST_API_v2" )
name_q = Q( entity_relation_trait__name = "coder_type" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
coder_type_q = name_q & value_q
relation_qs = relation_qs.filter( coder_type_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( coder_type_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [65]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
combined_q = pub_date_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
combined_q = combined_q & coder_username_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
combined_q = combined_q & coder_type_q
test_qs = relation_qs.filter( combined_q )
print( "\n\"&\" result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
test_qs = relation_qs.filter( pub_date_q )
test_qs = test_qs.filter( coder_username_q )
test_qs = test_qs.filter( coder_type_q )
print( "\n.filter() result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
combined_q = coder_username_q & coder_type_q
test_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
In [66]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
combined_q = pub_date_q | coder_username_q | coder_type_q
In [67]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "quoted" )
test_value_list.append( "mentioned" )
test_value_list.append( "shared_byline" )
relation_type_slug_q = Q( relation_type__slug__in = test_value_list )
combined_q = relation_type_slug_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( relation_type_slug_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [68]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
combined_q = Q( entity_trait__name = "sourcenet-Newspaper-ID" ) & Q( entity_trait__value = "1" )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [69]:
debug_flag = False
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# filter using IDs from above cell
#entity_qs = Entity.objects.filter( pk__in = entity_id_list )
# make Q()s
from_q = Q( relation_from__in = entity_qs )
to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# look for Entity_Relations to an Entity in any relation role (FROM, TO, THROUGH) that match Entity QS
# FROM
test_qs = relation_qs.filter( from_q )
print( "\nFROM - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# TO
test_qs = relation_qs.filter( to_q )
print( "\nTO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# THROUGH
test_qs = relation_qs.filter( through_q )
print( "\nTHROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO
combined_q = from_q | to_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# FROM | TO | THROUGH
combined_q = combined_q | through_q
test_qs = relation_qs.filter( combined_q )
print( "\nFROM | TO | THROUGH - result count: {} ( SQL: {} )".format( test_qs.count(), test_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( through_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
Back to Table of Contents
{
"comparison_type": "AND",
"filter_type": "AND",
"value_list": [
{
"comparison_type": "includes",
"filter_type": "relation_type_slug",
"relation_roles_list": [
"ALL"
],
"value_list": [
"mentioned",
"quoted",
"shared_byline"
]
},
{
"comparison_type": "AND",
"filter_type": "entity_type_slug",
"value_list": [
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"FROM"
],
"value": "person"
},
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"TO"
],
"value": "person"
},
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"THROUGH"
],
"value": "article"
}
]
},
{
"comparison_type": "AND",
"filter_type": "relation_trait",
"value_list": [
{
"comparison_type": "in_range",
"filter_type": "relation_trait",
"name": "pub_date",
"relation_roles_list": [
"ALL"
],
"value_from": "2009-12-01",
"value_to": "2009-12-31"
},
{
"comparison_type": "includes",
"filter_type": "relation_trait",
"name": "sourcenet-coder-User-username",
"relation_roles_list": [
"ALL"
],
"value_list": [
"automated"
]
},
{
"comparison_type": "includes",
"filter_type": "relation_trait",
"name": "coder_type",
"relation_roles_list": [
"ALL"
],
"value_list": [
"OpenCalais_REST_API_v2"
]
}
]
},
{
"comparison_type": "includes",
"filter_type": "entity_trait",
"name": "sourcenet-Newspaper-ID",
"relation_roles_list": [
"THROUGH"
],
"value_list": [
1
]
}
]
}
In [70]:
# start with all Entity_Relations.
#test_relation_qs = None
test_relation_qs = Entity_Relation.objects.all()
Back to Table of Contents
{
"comparison_type": "includes",
"filter_type": "relation_type_slug",
"relation_roles_list": [
"ALL"
],
"value_list": [
"mentioned",
"quoted",
"shared_byline"
]
},
In [71]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
#entity_qs = Entity.objects.filter( pk = 315 )
test_value_list = []
test_value_list.append( "quoted" )
test_value_list.append( "mentioned" )
test_value_list.append( "shared_byline" )
relation_type_slug_q = Q( relation_type__slug__in = test_value_list )
combined_q = relation_type_slug_q
relation_qs = relation_qs.filter( combined_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( relation_type_slug_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
Back to Table of Contents
{
"comparison_type": "AND",
"filter_type": "entity_type_slug",
"value_list": [
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"FROM"
],
"value": "person"
},
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"TO"
],
"value": "person"
},
{
"comparison_type": "equals",
"filter_type": "entity_type_slug",
"relation_roles_list": [
"THROUGH"
],
"value": "article"
}
]
},
In [72]:
# try to just find the entity with the desired identifiers.
# two QuerySets, one for just authors, and one for just articles.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
# Q for each of article and person entities.
article_entity_q = Q( entity_types__entity_type__slug = "article" )
person_entity_q = Q( entity_types__entity_type__slug = "person" )
# Make a QuerySet for each.
article_entity_qs = entity_qs.filter( article_entity_q )
print( "\narticle entity result count: {} (SQL: {})".format( article_entity_qs.count(), article_entity_qs.query ) )
person_entity_qs = entity_qs.filter( person_entity_q )
print( "\nperson entity result count: {} (SQL: {})".format( person_entity_qs.count(), person_entity_qs.query ) )
In [73]:
# make Q()s
from_q = Q( relation_from__in = person_entity_qs )
to_q = Q( relation_to__in = person_entity_qs )
through_q = Q( relation_through__in = article_entity_qs )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( from_q )
test_relation_qs = test_relation_qs.filter( to_q )
test_relation_qs = test_relation_qs.filter( through_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
Back to Table of Contents
{
"comparison_type": "AND",
"filter_type": "relation_trait",
"value_list": [
{
"comparison_type": "in_range",
"filter_type": "relation_trait",
"name": "pub_date",
"relation_roles_list": [
"ALL"
],
"value_from": "2009-12-01",
"value_to": "2009-12-31"
},
{
"comparison_type": "includes",
"filter_type": "relation_trait",
"name": "sourcenet-coder-User-username",
"relation_roles_list": [
"ALL"
],
"value_list": [
"automated"
]
},
{
"comparison_type": "includes",
"filter_type": "relation_trait",
"name": "coder_type",
"relation_roles_list": [
"ALL"
],
"value_list": [
"OpenCalais_REST_API_v2"
]
}
]
},
In [74]:
name_q = Q( entity_relation_trait__name = "pub_date" )
left_range_q = Q( entity_relation_trait__value__gte = "2009-12-01" )
right_range_q = Q( entity_relation_trait__value__lte = "2009-12-31" )
pub_date_q = name_q & left_range_q & right_range_q
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( pub_date_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [75]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
test_value_list = []
test_value_list.append( "automated" )
name_q = Q( entity_relation_trait__name = "sourcenet-coder-User-username" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
coder_username_q = name_q & value_q
relation_qs = relation_qs.filter( coder_username_q )
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( coder_username_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [76]:
# start with Entity_Relation QS
relation_qs = Entity_Relation.objects.all()
print( "\nresult count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
test_value_list = []
test_value_list.append( "OpenCalais_REST_API_v2" )
name_q = Q( entity_relation_trait__name = "coder_type" )
value_q = Q( entity_relation_trait__value__in = test_value_list )
coder_type_q = name_q & value_q
relation_qs = relation_qs.filter( coder_type_q )
print( "\ncoder_type result count: {} ( SQL: {} )".format( relation_qs.count(), relation_qs.query ) )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( coder_type_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [77]:
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( pub_date_q )
test_qs = test_qs.filter( coder_username_q )
test_qs = test_qs.filter( coder_type_q )
print( "all 3 together count: {}".format( test_qs.count() ) )
Back to Table of Contents
{
"comparison_type": "includes",
"filter_type": "entity_trait",
"name": "sourcenet-Newspaper-ID",
"relation_roles_list": [
"THROUGH"
],
"value_list": [
1
]
}
In [78]:
# try to just find the entity with the desired identifier.
entity_qs = Entity.objects.all()
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
combined_q = Q( entity_trait__name = "sourcenet-Newspaper-ID" ) & Q( entity_trait__value = "1" )
entity_qs = entity_qs.filter( combined_q )
print( "\nresult count: {} (SQL: {})".format( entity_qs.count(), entity_qs.query ) )
In [79]:
# make Q()s
#from_q = Q( relation_from__in = entity_qs )
#to_q = Q( relation_to__in = entity_qs )
through_q = Q( relation_through__in = entity_qs )
# are we doing an end-to-end test?
if ( test_relation_qs is not None ):
test_relation_qs = test_relation_qs.filter( through_q )
print( "\nend-to-end --> FROM & TO & THROUGH (filter) - result count: {} ( SQL: {} )".format( test_relation_qs.count(), test_relation_qs.query ) )
#-- END check to see if we are combining these filters. --#
In [80]:
# load basic file into test instance.
test_instance = TestHelper.load_basic()
#test_instance = TestHelper.load_basic_2()
In [81]:
# filter_specification
selection_filters = test_instance.get_selection_filters()
#print( "base_filter_spec: {}".format( base_filter_spec ) )
# retrieve actual "filter_specification".
filter_specification = selection_filters.get( NetworkDataRequest.PROP_NAME_FILTER_SPECIFICATION, None )
# load it into a FilterSpec.
filter_spec = FilterSpec()
filter_spec.set_filter_spec( filter_specification )
print( "filter_spec:\n{}".format( filter_spec.to_json_string() ) )
print( "- comparison type: {}".format( filter_spec.get_comparison_type() ) )
In [82]:
# build out Q() instances
result_status = test_instance.build_filter_spec_q( filter_spec )
print( "build Q() result: {}".format( result_status ) )
In [83]:
# try filtering.
relation_qs = Entity_Relation.objects.all()
relation_qs = test_instance.filter_relations_by_filter_spec( relation_qs, filter_spec )
print( "relation_qs count: {}".format( relation_qs.count() ) )
In [84]:
relation_qs = None
value_list = None
child_filter_spec_list = None
child_filter_spec_count = None
filter_1 = None
filter_1_q = None
test_qs = None
test_count = None
relation_count = None
# init QS
relation_qs = Entity_Relation.objects.all()
# work through child filter spec list
value_list = filter_spec.get_value_list()
child_filter_spec_list = filter_spec.get_child_filter_spec_list()
child_filter_spec_count = len( child_filter_spec_list )
print( "- Child filter spec count: {}".format( child_filter_spec_count ) )
# filter 1
filter_1 = child_filter_spec_list[ 0 ]
print( "Filter 1: {}\nJSON: {}".format( filter_1, filter_1.to_json_string() ) )
value_1 = value_list[ 0 ]
print( "Value 1: {}".format( value_1 ) )
# Q()
filter_1_q = filter_1.get_my_q()
print( "Q 1: {}".format( filter_1_q ) )
# test Q
test_qs = relation_qs.filter( filter_1_q )
test_count = test_qs.count()
print( "----> test_qs count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_1_q )
relation_count = relation_qs.count()
print( "\n\n- relation_qs count: {}".format( relation_count ) )
In [85]:
#relation_qs = None
current_q = None
test_qs = None
test_count = None
relation_count = None
filter_2 = None
filter_2_child_list = None
filter_2_child_1 = None
filter_2_child_1_q = None
filter_2_child_2 = None
filter_2_child_2_q = None
filter_2_child_3 = None
filter_2_child_3_q = None
# init QS
#relation_qs = Entity_Relation.objects.all()
# filter 2
filter_2 = child_filter_spec_list[ 1 ]
print( "Filter 2:\n{}".format( filter_2.to_json_string() ) )
# Q()
current_q = filter_2.get_my_q()
print( "Q 1: {}".format( current_q ) )
# children
filter_2_child_list = filter_2.get_child_filter_spec_list()
print( filter_2_child_list )
# child 1
filter_2_child_1 = filter_2_child_list[ 0 ]
print( "\n\nFilter 2, child 1:\n{}".format( filter_2_child_1.to_json_string() ) )
# child 1 Q
filter_2_child_1_q = filter_2_child_1.get_my_q()
print( "Q filter_2_child_1_q: {}".format( filter_2_child_1_q ) )
# test child 1 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_1_q )
test_count = test_qs.count()
print( "----> test_qs f2c1q count: {}".format( test_count ) )
# child 2
filter_2_child_2 = filter_2_child_list[ 1 ]
print( "\n\nFilter 2, child 2:\n{}".format( filter_2_child_2.to_json_string() ) )
# child 2 Q
filter_2_child_2_q = filter_2_child_2.get_my_q()
print( "Q filter_2_child_2_q: {}".format( filter_2_child_2_q ) )
# test child 2 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_2_q )
test_count = test_qs.count()
print( "----> test_qs f2c2q count: {}".format( test_count ) )
# child 3
filter_2_child_3 = filter_2_child_list[ 2 ]
print( "\n\nFilter 2, child 3:\n{}".format( filter_2_child_3.to_json_string() ) )
# child 3 Q
filter_2_child_3_q = filter_2_child_3.get_my_q()
print( "Q filter_2_child_3_q: {}".format( filter_2_child_3_q ) )
# test child 3 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_3_q )
test_count = test_qs.count()
print( "----> test_qs f2c3q count: {}".format( test_count ) )
# test all together
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_1_q )
test_qs = test_qs.filter( filter_2_child_2_q )
test_qs = test_qs.filter( filter_2_child_3_q )
test_count = test_qs.count()
print( "\n\n- test_qs ALL count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_2_child_1_q )
relation_qs = relation_qs.filter( filter_2_child_2_q )
relation_qs = relation_qs.filter( filter_2_child_3_q )
relation_count = relation_qs.count()
print( "- relation_qs count: {}".format( relation_count ) )
In [86]:
#relation_qs = None
current_q = None
test_qs = None
test_count = None
relation_count = None
filter_3 = None
filter_3_child_list = None
filter_3_child_1 = None
filter_3_child_1_q = None
filter_3_child_2 = None
filter_3_child_2_q = None
filter_3_child_3 = None
filter_3_child_3_q = None
# init QS
#relation_qs = Entity_Relation.objects.all()
# filter 3
filter_3 = child_filter_spec_list[ 2 ]
print( "Filter 3:\n{}".format( filter_3.to_json_string() ) )
# Q()
current_q = filter_3.get_my_q()
print( "Q 1: {}".format( current_q ) )
# children
filter_3_child_list = filter_3.get_child_filter_spec_list()
print( filter_3_child_list )
# child 1
filter_3_child_1 = filter_3_child_list[ 0 ]
print( "\n\nFilter 3, child 1:\n{}".format( filter_3_child_1.to_json_string() ) )
# child 1 Q
filter_3_child_1_q = filter_3_child_1.get_my_q()
print( "Q filter_3_child_1_q: {}".format( filter_3_child_1_q ) )
# test child 1 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_1_q )
test_count = test_qs.count()
print( "- test_qs f3c1q count: {}".format( test_count ) )
# child 2
filter_3_child_2 = filter_3_child_list[ 1 ]
print( "\n\nFilter 3, child 2:\n{}".format( filter_3_child_2.to_json_string() ) )
# child 2 Q
filter_3_child_2_q = filter_3_child_2.get_my_q()
print( "Q filter_3_child_2_q: {}".format( filter_3_child_2_q ) )
# test child 2 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_2_q )
test_count = test_qs.count()
print( "- test_qs f3c2q count: {}".format( test_count ) )
# child 3
filter_3_child_3 = filter_3_child_list[ 2 ]
print( "\n\nFilter 3, child 3:\n{}".format( filter_3_child_3.to_json_string() ) )
# child 3 Q
filter_3_child_3_q = filter_3_child_3.get_my_q()
print( "Q filter_3_child_3_q: {}".format( filter_3_child_3_q ) )
# test child 3 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_3_q )
test_count = test_qs.count()
print( "- test_qs f3c3q count: {}".format( test_count ) )
# test all together
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_1_q )
test_qs = test_qs.filter( filter_3_child_2_q )
test_qs = test_qs.filter( filter_3_child_3_q )
test_count = test_qs.count()
print( "\n\n- test_qs ALL count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_3_child_1_q )
relation_qs = relation_qs.filter( filter_3_child_2_q )
relation_qs = relation_qs.filter( filter_3_child_3_q )
relation_count = relation_qs.count()
print( "\n\n- relation_qs count: {}".format( relation_count ) )
In [87]:
filter_4 = None
value_4 = None
filter_4_q = None
test_qs = None
test_count = None
relation_count = None
# filter 4
filter_4 = child_filter_spec_list[ 3 ]
print( "Filter 4: {}\n- JSON: {}".format( filter_4, filter_4.to_json_string() ) )
value_4 = value_list[ 3 ]
print( "Value 4: {}".format( value_4 ) )
# Q()
filter_4_q = filter_4.get_my_q()
print( "Q 1: {}".format( filter_4_q ) )
# test Q
test_qs = relation_qs.filter( filter_4_q )
test_count = test_qs.count()
print( "- test_qs count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_4_q )
relation_count = relation_qs.count()
print( "- relation_qs count: {}".format( relation_count ) )
In [88]:
# load basic file into test instance.
#test_instance = TestHelper.load_basic()
test_instance = TestHelper.load_basic_2()
In [89]:
# filter_specification
selection_filters = test_instance.get_selection_filters()
#print( "base_filter_spec: {}".format( base_filter_spec ) )
# retrieve actual "filter_specification".
filter_specification = selection_filters.get( NetworkDataRequest.PROP_NAME_FILTER_SPECIFICATION, None )
# load it into a FilterSpec.
filter_spec = FilterSpec()
filter_spec.set_filter_spec( filter_specification )
print( "filter_spec:\n{}".format( filter_spec.to_json_string() ) )
print( "- comparison type: {}".format( filter_spec.get_comparison_type() ) )
In [90]:
# build out Q() instances
result_status = test_instance.build_filter_spec_q( filter_spec )
print( "build Q() result: {}".format( result_status ) )
In [91]:
# try filtering.
relation_qs = Entity_Relation.objects.all()
relation_qs = test_instance.filter_relations_by_filter_spec( relation_qs, filter_spec )
print( "relation_qs count: {}".format( relation_qs.count() ) )
In [92]:
relation_qs = None
value_list = None
child_filter_spec_list = None
child_filter_spec_count = None
filter_1 = None
filter_1_q = None
test_qs = None
test_count = None
relation_count = None
# init QS
relation_qs = Entity_Relation.objects.all()
# work through child filter spec list
value_list = filter_spec.get_value_list()
child_filter_spec_list = filter_spec.get_child_filter_spec_list()
child_filter_spec_count = len( child_filter_spec_list )
print( "- Child filter spec count: {}".format( child_filter_spec_count ) )
# filter 1
filter_1 = child_filter_spec_list[ 0 ]
print( "Filter 1: {}\nJSON: {}".format( filter_1, filter_1.to_json_string() ) )
value_1 = value_list[ 0 ]
print( "Value 1: {}".format( value_1 ) )
# Q()
filter_1_q = filter_1.get_my_q()
print( "Q 1: {}".format( filter_1_q ) )
# test Q
test_qs = relation_qs.filter( filter_1_q )
test_count = test_qs.count()
print( "----> test_qs count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_1_q )
relation_count = relation_qs.count()
print( "\n\n- relation_qs count: {}".format( relation_count ) )
In [93]:
#relation_qs = None
current_q = None
test_qs = None
test_count = None
relation_count = None
filter_2 = None
filter_2_child_list = None
filter_2_child_1 = None
filter_2_child_1_q = None
filter_2_child_2 = None
filter_2_child_2_q = None
filter_2_child_3 = None
filter_2_child_3_q = None
# init QS
#relation_qs = Entity_Relation.objects.all()
# filter 2
filter_2 = child_filter_spec_list[ 1 ]
print( "Filter 2:\n{}".format( filter_2.to_json_string() ) )
# Q()
current_q = filter_2.get_my_q()
print( "Q 1: {}".format( current_q ) )
# children
filter_2_child_list = filter_2.get_child_filter_spec_list()
print( filter_2_child_list )
# child 1
filter_2_child_1 = filter_2_child_list[ 0 ]
print( "\n\nFilter 2, child 1:\n{}".format( filter_2_child_1.to_json_string() ) )
# child 1 Q
filter_2_child_1_q = filter_2_child_1.get_my_q()
print( "Q filter_2_child_1_q: {}".format( filter_2_child_1_q ) )
# test child 1 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_1_q )
test_count = test_qs.count()
print( "----> test_qs f2c1q count: {}".format( test_count ) )
# child 2
filter_2_child_2 = filter_2_child_list[ 1 ]
print( "\n\nFilter 2, child 2:\n{}".format( filter_2_child_2.to_json_string() ) )
# child 2 Q
filter_2_child_2_q = filter_2_child_2.get_my_q()
print( "Q filter_2_child_2_q: {}".format( filter_2_child_2_q ) )
# test child 2 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_2_q )
test_count = test_qs.count()
print( "----> test_qs f2c2q count: {}".format( test_count ) )
# child 3
filter_2_child_3 = filter_2_child_list[ 2 ]
print( "\n\nFilter 2, child 3:\n{}".format( filter_2_child_3.to_json_string() ) )
# child 3 Q
filter_2_child_3_q = filter_2_child_3.get_my_q()
print( "Q filter_2_child_3_q: {}".format( filter_2_child_3_q ) )
# test child 3 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_3_q )
test_count = test_qs.count()
print( "----> test_qs f2c3q count: {}".format( test_count ) )
# test all together
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_2_child_1_q )
test_qs = test_qs.filter( filter_2_child_2_q )
test_qs = test_qs.filter( filter_2_child_3_q )
test_count = test_qs.count()
print( "\n\n- test_qs ALL count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_2_child_1_q )
relation_qs = relation_qs.filter( filter_2_child_2_q )
relation_qs = relation_qs.filter( filter_2_child_3_q )
relation_count = relation_qs.count()
print( "- relation_qs count: {}".format( relation_count ) )
In [94]:
#relation_qs = None
current_q = None
test_qs = None
test_count = None
relation_count = None
filter_3 = None
filter_3_child_list = None
filter_3_child_1 = None
filter_3_child_1_q = None
filter_3_child_2 = None
filter_3_child_2_q = None
filter_3_child_3 = None
filter_3_child_3_q = None
# init QS
#relation_qs = Entity_Relation.objects.all()
# filter 3
filter_3 = child_filter_spec_list[ 2 ]
print( "Filter 3:\n{}".format( filter_3.to_json_string() ) )
# Q()
current_q = filter_3.get_my_q()
print( "Q 1: {}".format( current_q ) )
# children
filter_3_child_list = filter_3.get_child_filter_spec_list()
print( filter_3_child_list )
# child 1
filter_3_child_1 = filter_3_child_list[ 0 ]
print( "\n\nFilter 3, child 1:\n{}".format( filter_3_child_1.to_json_string() ) )
# child 1 Q
filter_3_child_1_q = filter_3_child_1.get_my_q()
print( "Q filter_3_child_1_q: {}".format( filter_3_child_1_q ) )
# test child 1 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_1_q )
test_count = test_qs.count()
print( "- test_qs f3c1q count: {}".format( test_count ) )
# child 2
filter_3_child_2 = filter_3_child_list[ 1 ]
print( "\n\nFilter 3, child 2:\n{}".format( filter_3_child_2.to_json_string() ) )
# child 2 Q
filter_3_child_2_q = filter_3_child_2.get_my_q()
print( "Q filter_3_child_2_q: {}".format( filter_3_child_2_q ) )
# test child 2 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_2_q )
test_count = test_qs.count()
print( "- test_qs f3c2q count: {}".format( test_count ) )
# child 3
filter_3_child_3 = filter_3_child_list[ 2 ]
print( "\n\nFilter 3, child 3:\n{}".format( filter_3_child_3.to_json_string() ) )
# child 3 Q
filter_3_child_3_q = filter_3_child_3.get_my_q()
print( "Q filter_3_child_3_q: {}".format( filter_3_child_3_q ) )
# test child 3 Q
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_3_q )
test_count = test_qs.count()
print( "- test_qs f3c3q count: {}".format( test_count ) )
# test all together
test_qs = Entity_Relation.objects.all()
test_qs = test_qs.filter( filter_3_child_1_q )
test_qs = test_qs.filter( filter_3_child_2_q )
test_qs = test_qs.filter( filter_3_child_3_q )
test_count = test_qs.count()
print( "\n\n- test_qs ALL count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_3_child_1_q )
relation_qs = relation_qs.filter( filter_3_child_2_q )
relation_qs = relation_qs.filter( filter_3_child_3_q )
relation_count = relation_qs.count()
print( "\n\n- relation_qs count: {}".format( relation_count ) )
In [95]:
filter_4 = None
value_4 = None
filter_4_q = None
test_qs = None
test_count = None
relation_count = None
# filter 4
filter_4 = child_filter_spec_list[ 3 ]
print( "Filter 4: {}\n- JSON: {}".format( filter_4, filter_4.to_json_string() ) )
value_4 = value_list[ 3 ]
print( "Value 4: {}".format( value_4 ) )
# Q()
filter_4_q = filter_4.get_my_q()
print( "Q 1: {}".format( filter_4_q ) )
# test Q
test_qs = relation_qs.filter( filter_4_q )
test_count = test_qs.count()
print( "- test_qs count: {}".format( test_count ) )
# update QS
relation_qs = relation_qs.filter( filter_4_q )
relation_count = relation_qs.count()
print( "- relation_qs count: {}".format( relation_count ) )
Network data filtering TODO:
design JSON based on https://research.local/research/context/text/output/network: generalized, but not perfectly so. Remaining tasks (can be deferred for now):
filter, same for entities and relations:
specify the output:
if file:
build logic to use contents of JSON to filter QuerySets of Entities and Relations, then will need to make lots more tests.
testing:
FilterSpec
NetworkDataRequest - start with low-level filter-type-specific methods and work your way up.
compact_entity_relation_queryset()"general TODO:
methods to find relations, similar to filter_entities() and lookup_entities() in Entity model class. Include:
from_entity_traits
// to =
to_entity_traits
// through =
through_entity_traits
either FROM or TO (so undirected search - "I don't care which side")
any of FROM, TO, THROUGH
relation_traits
NOTE for trait matching types (and probably entity identifiers, also):
Tags on Entity and/or Entity_Relation.
make small dummy person and organization classes in context, so I can use them to test Abstract_Entity_Container without needing context_text.
come up with better way to seed entities and relations for sourcenet - store spec in context_text base, then method to create or update all.
This will need to build up a basic object mapping based on foriegn keys before it creates anything, then create things in the right order (Entity_Types and related first, then Entity_Identifier_Types, then Relation_Types). Order:
for each type:
To actually load, go in order of types outlined above, creating as you go.
abstraction:
make an abstract parent for a type that has associated trait specs (parent to Entity_Type and Entity_Relation_Type).
get_trait_spec().make an abstract parent for trait containers that have associated types with associated trait specs (parent to Entity and Entity_Relation).
Network data filtering TODO:
2019.12.10:
2019.12.13 - design JSON based on https://research.local/research/context/text/output/network: generalized, but not perfectly so:
filter, same for entities and relations:
// trait-based filters (allow for
// in general - per trait:
trait values (set appropriate depending on type of comparison):
// pub_date ranges - list of one or more pairs of pub-date start and end date ranges.
// coder_types to include or exclude.
// identifier-based filters...
specify the output:
if file:
// data format - choose from:
// output types:
// include headers?
2019.12.19 - add "filter_type" as a property in filter criteria, make "AND" and "OR" filter types, and then just have a filter specification, don't have the 5 separate lists.
build logic to use contents of JSON to filter QuerySets of Entities and Relations, then will need to make lots more tests.
testing:
// FilterSpec
NetworkDataRequest - start with low-level filter-type-specific methods and work your way up.
low-level methods for actually filtering on each type of filter, based on a single filter spec:
2019.12.20 - methods for tying the different filter specs and filter types together:
// build_filter_spec_aggregate_q (just handles comparison types AND and OR)
// build_filter_spec_q
2020.01.06
design JSON based on https://research.local/research/context/text/output/network: generalized, but not perfectly so. Remaining tasks (can be deferred for now):
specify the output:
build logic to use contents of JSON to filter QuerySets of Entities and Relations, then will need to make lots more tests.
testing:
NetworkDataRequest - start with low-level filter-type-specific methods and work your way up.
// methods for coordinating the overall filter plan stored in the request.
// filter_relation_query_set()
// filter_relations()