notebook.community

Edit and run 





In [1]:



    


from part_three_meta import wrapping_test_path
import pytest
from pprint import pprint

Dynamic and programmatic extension of mongoengine.Document class mapped to existing MongoDB documents

…continued from part 02

Immediate goal: implement an object-document mapper called MongoEngine to facilitate building a RESTful application programming interface (API) for data stored in a MongoDB database.

End goal: display data using ReactJS compontent Grilldle

More steps to get the data into Griddle.

One way to get the data I want to display into Griddle is via a RESTful application programming interface(API).

I tried a simple instance of FlaskRESTful.

I was able to build a simple API though had some serialization issues returning data directly from the Mongo database.

This is the code that led to the serialization issues. Using a different serializer solved the issue. I suspected that there was probably a better way.

Some more investigation led me to believe that using an object-document mapper such as MongoEngine would make building a RESTful API easier.

So I wrote some code to extend mongoengine.Document

I followed the MongoEngine documentation here for extending the Document class.

The code I wrote to extend mongoengine.Document is here..

Problems that led me to try to programmatically extend mongoengine.Document rather than type it all out by hand using the class keyword:

The code I wrote to (almost) solve those problems.

View the code here on GitHub.

The code is not easy to read and is the result of testing and tweaking the list comprehensions until I had the result I wanted. This is preferable because adding the attributes to an extended mongoengine.Document by hand would have been a nightmare to manage.

The method I used is programmatic and predictable and best of all easily testable. (Current tests are failing and therefore are revealing information I did not know and could not have known otherwise!)

Two new classes are created: Accident and Citation. These classes are dynamically created based on the existing attributes of their respective documents in existing MongoDB data collections.





In [3]:



    


# This cell is meta code whose purpose is to display HTML into this notebook 
# so that it will display in the Zippped Code blog.
import html
from IPython.core.display import display, HTML
from flpd_helper.documents import namespaces, convert_namespace, new_collection_names
from html_tables import get_table_from, new_collection_name_template, scroll_message, two_tables_title
from collections import OrderedDict

display(HTML(two_tables_title))
for namespace, new_collection_name, class_name in zip(namespaces, new_collection_names, ('Accident', 'Citation', )):
    # convert for displaying in this notebook table
    namespace = tuple(sorted([((key, "invalid: '{}' valid: '{}'".format(*key),), value) 
                              for key, value in namespace.items()]))
    namespace = OrderedDict(namespace)
    dictionary = convert_namespace(namespace, new_collection_name)
    dictionary = tuple(sorted([(key, html.escape(repr(value))) for key, value in dictionary.items()]))
    dictionary = OrderedDict(dictionary)
    display(HTML(new_collection_name_template.format(class_name)))
    display(HTML(scroll_message))
    display(get_table_from(dictionary))



    


















    







These 2 tables show the new dynamically created classes' attributes. The classes "Accident" and "Citation" are extended mongoengine.Documents.










    







'Accident' with existing invalid Python identifiers, converted valid Python identifiers and mapped programmatically mapped mongoengine.field classes.










    






You may have to scroll the table left and right to see all the data.









    







invalid: '1st&nbspHarmful&nbspEvent&nbspCode'&nbspvalid: '_1st_harmful_event_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c07f0>
invalid: '1st&nbspHarmful&nbspEvent&nbspDescription'&nbspvalid: '_1st_harmful_event_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0828>
invalid: 'Access&nbspControl&nbspCode'&nbspvalid: 'access_control_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c0860>
invalid: 'Accident&nbspDescription'&nbspvalid: 'accident_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0898>
invalid: 'Agency&nbspCode'&nbspvalid: 'agency_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c08d0>
invalid: 'Ambulance&nbspRequest&nbspDescription'&nbspvalid: 'ambulance_request_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0908>
invalid: 'At&nbspIntersection&nbspRoad&nbspType&nbspCode'&nbspvalid: 'at_intersection_road_type_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0940>
invalid: 'At&nbspIntersection&nbspRoad&nbspType&nbspDesc'&nbspvalid: 'at_intersection_road_type_desc'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0978>
invalid: 'Case&nbspNumber'&nbspvalid: 'case_number'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c09b0>
invalid: 'City'&nbspvalid: 'city'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0a20>
invalid: 'Corrected&nbspReport'&nbspvalid: 'corrected_report'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0a58>
invalid: 'County'&nbspvalid: 'county'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0ac8>
invalid: 'Date/Time&nbspOccurred'&nbspvalid: 'datetime_occurred'<mongoengine.fields.DateTimeField&nbspobject&nbspat&nbsp0x7fcb637c0b00>
invalid: 'Day&nbspof&nbspWeek'&nbspvalid: 'day_of_week'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0b38>
invalid: 'Development&nbspType&nbspCode'&nbspvalid: 'development_type_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0b70>
invalid: 'Development&nbspType&nbspDescription'&nbspvalid: 'development_type_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0ba8>
invalid: 'Direction&nbspFrom'&nbspvalid: 'direction_from'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0be0>
invalid: 'Direction&nbspToward'&nbspvalid: 'direction_toward'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0c18>
invalid: 'District&nbspCode'&nbspvalid: 'district_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0c50>
invalid: 'District&nbspDescription'&nbspvalid: 'district_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0c88>
invalid: 'Employee&nbspDivision'&nbspvalid: 'employee_division'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0cc0>
invalid: 'Employee&nbspSection'&nbspvalid: 'employee_section'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0cf8>
invalid: 'Employee&nbspShift'&nbspvalid: 'employee_shift'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0d30>
invalid: 'Employee&nbspUnit'&nbspvalid: 'employee_unit'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0d68>
invalid: 'Feet&nbspFrom'&nbspvalid: 'feet_from'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0da0>
invalid: 'Harmful&nbspEvent&nbspCode'&nbspvalid: 'harmful_event_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c0dd8>
invalid: 'Hazmat&nbspInvolved'&nbspvalid: 'hazmat_involved'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0e10>
invalid: 'In/Near'&nbspvalid: 'innear'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0ef0>
invalid: 'Injured&nbspTaken&nbspTo&nbspCode'&nbspvalid: 'injured_taken_to_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0e80>
invalid: 'Injured&nbspTaken&nbspTo&nbspDescription'&nbspvalid: 'injured_taken_to_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0eb8>
invalid: 'Intersection'&nbspvalid: 'intersection'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0f28>
invalid: 'Light&nbspCondition&nbspCode'&nbspvalid: 'light_condition_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c0f60>
invalid: 'Light&nbspCondition&nbspDescription'&nbspvalid: 'light_condition_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0f98>
invalid: 'Locality&nbspCode'&nbspvalid: 'locality_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c0fd0>
invalid: 'Locality&nbspDescription'&nbspvalid: 'locality_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7048>
invalid: 'Location&nbspof&nbspImpact&nbspCode'&nbspvalid: 'location_of_impact_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7080>
invalid: 'Location&nbspof&nbspImpact&nbspDescription'&nbspvalid: 'location_of_impact_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c70b8>
invalid: 'Manner&nbspof&nbspCollision&nbspCode'&nbspvalid: 'manner_of_collision_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c70f0>
invalid: 'Manner&nbspof&nbspCollision&nbspor&nbspImpact'&nbspvalid: 'manner_of_collision_or_impact'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7128>
invalid: 'Neighborhood&nbspCode'&nbspvalid: 'neighborhood_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7160>
invalid: 'Neighborhood&nbspDescription'&nbspvalid: 'neighborhood_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7198>
invalid: 'Next&nbspReference&nbspPoint&nbspCode'&nbspvalid: 'next_reference_point_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c71d0>
invalid: 'Next&nbspReference&nbspPoint&nbspDescript'&nbspvalid: 'next_reference_point_descript'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7208>
invalid: 'Non&nbspIntersection&nbspRoad&nbspType&nbspDescript'&nbspvalid: 'non_intersection_road_type_descript'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7240>
invalid: 'Notified&nbspby&nbspDriver&nbspor&nbspLaw&nbspEnforcement'&nbspvalid: 'notified_by_driver_or_law_enforcement'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7278>
invalid: 'Notn&nbspIntersection&nbspRoad&nbspType&nbspCode'&nbspvalid: 'notn_intersection_road_type_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c72b0>
invalid: 'Number&nbspof&nbspLanes'&nbspvalid: 'number_of_lanes'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c72e8>
invalid: 'Number&nbspof&nbspUnits&nbspInvolved'&nbspvalid: 'number_of_units_involved'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7320>
invalid: 'Officer&nbspDept&nbspID'&nbspvalid: 'officer_dept_id'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7358>
invalid: 'Pedestrian&nbspControl'&nbspvalid: 'pedestrian_control'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7390>
invalid: 'Private&nbspProperty'&nbspvalid: 'private_property'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c73c8>
invalid: 'Relation&nbspto&nbspjunction'&nbspvalid: 'relation_to_junction'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7400>
invalid: 'Report&nbspArea&nbspCode'&nbspvalid: 'report_area_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7438>
invalid: 'Report&nbspArea&nbspDescription'&nbspvalid: 'report_area_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7470>
invalid: 'Road&nbspCharacter&nbspDescription'&nbspvalid: 'road_character_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c74e0>
invalid: 'Road&nbspCharacter'&nbspvalid: 'road_character'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c74a8>
invalid: 'Road&nbspClass&nbspDescription'&nbspvalid: 'road_class_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7550>
invalid: 'Road&nbspClass'&nbspvalid: 'road_class'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7518>
invalid: 'Road&nbspDefects&nbspCode'&nbspvalid: 'road_defects_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7588>
invalid: 'Road&nbspDefects&nbspDescription'&nbspvalid: 'road_defects_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c75c0>
invalid: 'Road&nbspFeature&nbspCode'&nbspvalid: 'road_feature_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c75f8>
invalid: 'Road&nbspSurface&nbspCodes'&nbspvalid: 'road_surface_codes'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7630>
invalid: 'Road&nbspSurface&nbspConditions'&nbspvalid: 'road_surface_conditions'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7668>
invalid: 'Road&nbspSurface&nbspDescription'&nbspvalid: 'road_surface_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c76a0>
invalid: 'Road&nbspType&nbspCode'&nbspvalid: 'road_type_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c76d8>
invalid: 'Road&nbspconstruction'&nbspvalid: 'road_construction'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7710>
invalid: 'School&nbspBus&nbspRelated'&nbspvalid: 'school_bus_related'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7748>
invalid: 'Street'&nbspvalid: 'street'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c77b8>
invalid: 'Subdivision&nbspDescription'&nbspvalid: 'subdivision_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7828>
invalid: 'Subdivision'&nbspvalid: 'subdivision'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c77f0>
invalid: 'Supplement&nbspReport'&nbspvalid: 'supplement_report'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7860>
invalid: 'Taken&nbspBy&nbspCode'&nbspvalid: 'taken_by_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7898>
invalid: 'Time&nbspCleared&nbspScene'&nbspvalid: 'time_cleared_scene'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c78d0>
invalid: 'Time&nbspOccured'&nbspvalid: 'time_occured'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7908>
invalid: 'Time&nbspReported'&nbspvalid: 'time_reported'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7940>
invalid: 'Township&nbspCode'&nbspvalid: 'township_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7978>
invalid: 'Township&nbspDescription'&nbspvalid: 'township_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c79b0>
invalid: 'Tract&nbspCode'&nbspvalid: 'tract_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c79e8>
invalid: 'Tract&nbspDescription'&nbspvalid: 'tract_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7a20>
invalid: 'Trafficway&nbspFlow&nbspCode'&nbspvalid: 'trafficway_flow_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7a58>
invalid: 'Trafficway&nbspFlow&nbspDescription'&nbspvalid: 'trafficway_flow_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7a90>
invalid: 'Visible&nbspCode'&nbspvalid: 'visible_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7ac8>
invalid: 'Visible&nbspDescription'&nbspvalid: 'visible_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7b00>
invalid: 'Weather&nbspCode'&nbspvalid: 'weather_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7b38>
invalid: 'Weather&nbspDescription'&nbspvalid: 'weather_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7b70>
invalid: 'Within&nbspInterchange'&nbspvalid: 'within_interchange'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7ba8>
invalid: 'Zone&nbspCode'&nbspvalid: 'zone_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7be0>
invalid: 'Zone&nbspDescription'&nbspvalid: 'zone_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7c18>
invalid: 'dnr'&nbspvalid: 'dnr'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7c88>
invalid: 'fatality'&nbspvalid: 'fatality'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7cc0>
invalid: 'geoother'&nbspvalid: 'geoother'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7cf8>
invalid: 'geox'&nbspvalid: 'geox'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7d68>
invalid: 'geoy'&nbspvalid: 'geoy'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7dd8>
invalid: 'interramp'&nbspvalid: 'interramp'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7e10>
invalid: 'location'&nbspvalid: 'location'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7e48>
invalid: 'other'&nbspvalid: 'other'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7eb8>
invalid: 'recCount'&nbspvalid: 'reccount'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7ef0>
invalid: 'Accident&nbspNumber'&nbspvalid: 'accident_number'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637c7f28>
meta{'collection': 'mapped_accidents'}










    







'Citation' with existing invalid Python identifiers, converted valid Python identifiers and mapped programmatically mapped mongoengine.field classes.










    






You may have to scroll the table left and right to see all the data.









    







invalid: 'Apartment/Floor'&nbspvalid: 'apartmentfloor'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7f60>
invalid: 'Case&nbspNumber'&nbspvalid: 'case_number'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7f98>
invalid: 'Citation&nbspCharge'&nbspvalid: 'citation_charge'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637c7fd0>
invalid: 'Citation&nbspDelivered&nbspto&nbspPerson'&nbspvalid: 'citation_delivered_to_person'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc048>
invalid: 'Court&nbspAddress'&nbspvalid: 'court_address'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc080>
invalid: 'Court&nbspCode'&nbspvalid: 'court_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc0b8>
invalid: 'Court&nbspDate/Time'&nbspvalid: 'court_datetime'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc0f0>
invalid: 'Court&nbspName'&nbspvalid: 'court_name'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc128>
invalid: 'Crash'&nbspvalid: 'crash'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc198>
invalid: 'Date&nbspOccurred'&nbspvalid: 'date_occurred'<mongoengine.fields.DateTimeField&nbspobject&nbspat&nbsp0x7fcb637cc1d0>
invalid: 'Detection&nbspDevice'&nbspvalid: 'detection_device'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc208>
invalid: 'District&nbspCode'&nbspvalid: 'district_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc240>
invalid: 'District&nbspDescription'&nbspvalid: 'district_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc278>
invalid: 'Driving&nbspwith&nbspUnlawful&nbspBlood&nbspAlcohol&nbspLevel'&nbspvalid: 'driving_with_unlawful_blood_alcohol_level'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc2b0>
invalid: 'Employee&nbspDivision&nbspCode'&nbspvalid: 'employee_division_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc2e8>
invalid: 'Employee&nbspSection&nbspCode'&nbspvalid: 'employee_section_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc320>
invalid: 'Employee&nbspShift&nbspCode'&nbspvalid: 'employee_shift_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc358>
invalid: 'Employee&nbspUnit&nbspCode'&nbspvalid: 'employee_unit_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc390>
invalid: 'Ethnicity&nbspCode'&nbspvalid: 'ethnicity_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc3c8>
invalid: 'Neighborhood&nbspCode'&nbspvalid: 'neighborhood_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc400>
invalid: 'Neighborhood&nbspDescription'&nbspvalid: 'neighborhood_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc438>
invalid: 'Police&nbspAgency&nbspCode'&nbspvalid: 'police_agency_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc470>
invalid: 'Police&nbspAgency&nbspName'&nbspvalid: 'police_agency_name'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc4a8>
invalid: 'Race&nbspCode'&nbspvalid: 'race_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc4e0>
invalid: 'Reporting&nbspArea&nbspCode'&nbspvalid: 'reporting_area_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc518>
invalid: 'Reporting&nbspArea&nbspDesc'&nbspvalid: 'reporting_area_desc'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc550>
invalid: 'Residence&nbspCode'&nbspvalid: 'residence_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc588>
invalid: 'Sex&nbspCode'&nbspvalid: 'sex_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc5c0>
invalid: 'Street&nbspDirection'&nbspvalid: 'street_direction'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc5f8>
invalid: 'Street&nbspName'&nbspvalid: 'street_name'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc630>
invalid: 'Street&nbspNumber'&nbspvalid: 'street_number'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc668>
invalid: 'Subdivision&nbspCode'&nbspvalid: 'subdivision_code'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc6a0>
invalid: 'Subdivision&nbspDescription'&nbspvalid: 'subdivision_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc6d8>
invalid: 'Time&nbspEntered'&nbspvalid: 'time_entered'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc710>
invalid: 'Tract&nbspDescription'&nbspvalid: 'tract_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc748>
invalid: 'Zone&nbspCode'&nbspvalid: 'zone_code'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc780>
invalid: 'Zone&nbspDescription'&nbspvalid: 'zone_description'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc7b8>
invalid: 'age'&nbspvalid: 'age'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cc828>
invalid: 'city'&nbspvalid: 'city'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc898>
invalid: 'geox'&nbspvalid: 'geox'<mongoengine.fields.FloatField&nbspobject&nbspat&nbsp0x7fcb637cc908>
invalid: 'geoy'&nbspvalid: 'geoy'<mongoengine.fields.FloatField&nbspobject&nbspat&nbsp0x7fcb637cc978>
invalid: 'location'&nbspvalid: 'location'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cc9b0>
invalid: 'speed'&nbspvalid: 'speed'<mongoengine.fields.IntField&nbspobject&nbspat&nbsp0x7fcb637cca20>
invalid: 'tract'&nbspvalid: 'tract'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637cca90>
invalid: 'Citation&nbspNumber'&nbspvalid: 'citation_number'<mongoengine.fields.StringField&nbspobject&nbspat&nbsp0x7fcb637ccac8>
meta{'collection': 'mapped_citations'}

I tested the dynamically extended documents Accident and Citation and discovered this.

>           raise ValidationError(message, errors=errors)
E           mongoengine.errors.ValidationError: ValidationError (Accident:None) (StringField only accepts string values: ['district_code', 'zone_code', 'report_area_description', 'geox', 'geoy'])

The test revealed that the sample data I used to map the mongoengine fields had mixtures of str and int or float values. So when I tried to create document instances from the object-document mapper document, the validation tests failed.

I am beginning to really appreciate the value of an object-document mapper!

If you look in the tables above, for example, 'geox' has a string type in the first table and a float type in the second table. This is because the sample against which I mapped had an empty string as a value. In the second table, a valid value existed so the expected mongoengin.fields field class was inserted.

So then I wrote some code to filter which values had multiple data types.

After a session of trial and error in this Jupyter notebook I was able to write code in the form of a complex list comprehensions that filtered out which fields in the database had multiple data types. The end result was a mapping from which a default and sane mongoengine field can be retrieved even when the values have more than one date type.

The code is complex. The trade-off is that I do not have to manually type the extended classes using the class keyword. The number of attributes is unwieldy so it is better to write testable code that can manage extension dynamically using Python's built-in type.

Next…

Using the new lookup map to cast disparate data types among values of the same fields in documents of the same collection.

Content source: dm-wyncode/zipped-code

Similar notebooks:

notebook.community | gallery | about