The pipelines are run on the server periodically and based on pipeline and data dependencies.
You can also run specific pipelines manually for development or to run custom pipelines.
In [1]:
import os
os.chdir('..')
os.getcwd()
Out[1]:
'/pipelines'
In [2]:
!{'dpp'}
Available Pipelines:
- ./knesset/kns_knessetdates (*)
- ./knesset/kns_govministry (*)
- ./knesset/kns_itemtype (*)
- ./knesset/kns_status (*)
- ./committees/kns_committee (*)
- ./committees/kns_jointcommittee (*)
- ./committees/kns_cmtsitecode (*)
- ./committees/kns_cmtsessionitem (*)
- ./committees/kns_documentcommitteesession (*)
- ./committees/sync-documents (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_documentcommitteesession
Missing dependency: Couldn't open datapackage data/committees/kns_documentcommitteesession/datapackage.json
- ./committees/kns_committeesession (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committee
Missing dependency: Couldn't open datapackage data/committees/kns_committee/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_cmtsessionitem
Missing dependency: Couldn't open datapackage data/committees/kns_cmtsessionitem/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_documentcommitteesession
Missing dependency: Couldn't open datapackage data/committees/kns_documentcommitteesession/datapackage.json
- ./committees/dist/copy_static_files (*)
- ./votes_kmember/join_votes_shadow_mk (*)
- ./lobbyists/v_lobbyist (*)
- ./lobbyists/v_lobbyist_clients (*)
- ./people/attendance/govministries-managers (*)
- ./votes/view_vote_mk_individual (*)
- ./votes/vote_result_type (*)
- ./laws/kns_law (*)
- ./laws/kns_law_binding (*)
- ./laws/kns_document_law (*)
- ./laws/kns_israel_law (*)
- ./laws/kns_israel_law_name (*)
- ./laws/kns_israel_law_ministry (*)
- ./laws/kns_israel_law_classification (*)
- ./laws/kns_israel_law_binding (*)
- ./plenum/kns_plenumsession (*)
- ./plenum/kns_plmsessionitem (*)
- ./plenum/kns_documentplenumsession (*)
- ./members/kns_person (*)
- ./members/kns_position (*)
- ./members/kns_persontoposition (*)
- ./members/kns_mksitecode (*)
- ./members/mk_individual (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/kns_mksitecode
Missing dependency: Couldn't open datapackage data/members/kns_mksitecode/datapackage.json
Missing dependency: Couldn't open datapackage data/members/kns_persontoposition/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/kns_position
Missing dependency: Couldn't open datapackage data/members/kns_position/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/kns_person
Missing dependency: Couldn't open datapackage data/members/kns_person/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
- ./members/presence (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./bills/kns_bill (*)
- ./bills/kns_billname (*)
- ./bills/kns_billinitiator (*)
- ./bills/kns_billhistoryinitiator (*)
- ./bills/kns_billsplit (*)
- ./bills/kns_billunion (*)
- ./bills/kns_documentbill (*)
- ./committees/background_material_titles (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_documentcommitteesession
Missing dependency: Couldn't open datapackage data/committees/kns_documentcommitteesession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committeesession
Missing dependency: Couldn't open datapackage data/committees/kns_committeesession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committee
Missing dependency: Couldn't open datapackage data/committees/kns_committee/datapackage.json
- ./committees/dist/build_positions (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./committees/dist/create_factions (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/build_positions
Missing dependency: Couldn't open datapackage data/committees/dist/positions_aggr/datapackage.json
- ./people/source-data-stats (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committeesession
Missing dependency: Couldn't open datapackage data/committees/kns_committeesession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./people/committee-meeting-attendees (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committeesession
Missing dependency: Couldn't open datapackage data/committees/kns_committeesession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./people/committee-meeting-attendees-mks-stats (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
- ./people/committee-meeting-attendees-mks-full-stats (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committeesession
Missing dependency: Couldn't open datapackage data/committees/kns_committeesession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
- ./people/attendance/committee-meetings (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./people/attendance/committee-meetings-export (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./people/attendance/committee-meetings
Missing dependency: Couldn't open datapackage data/people/attendance/committee_meetings/datapackage.json
- ./people/attendance/committee-meetings-govministries (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./votes/vote_rslts_kmmbr_shadow (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./votes/vote_result_type
Missing dependency: Couldn't open datapackage data/votes/vote_result_type/datapackage.json
- ./web_ui/meetings (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committee
Missing dependency: Couldn't open datapackage data/committees/kns_committee/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./committees/dist/build (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/kns_committee
Missing dependency: Couldn't open datapackage data/committees/kns_committee/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./committees/dist/render_meetings (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/build
Missing dependency: Couldn't open datapackage data/committees/dist/build_meetings/datapackage.json
- ./committees/dist/render_committees (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/render_meetings
Missing dependency: Couldn't open datapackage data/committees/dist/rendered_meetings_stats/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/build
Missing dependency: Couldn't open datapackage data/committees/dist/build_meetings/datapackage.json
- ./committees/dist/create_members (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./people/committee-meeting-attendees
Missing dependency: Couldn't open datapackage data/people/committees/meeting-attendees/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
- ./committees/dist/sync-dist (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/render_meetings
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/render_committees
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/create_members
Dirty dependency: Cannot run until dependency is executed: ./committees/dist/create_factions
- ./votes/view_vote_rslts_hdr_approved (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./votes/vote_rslts_kmmbr_shadow
Missing dependency: Couldn't open datapackage data/votes/vote_rslts_kmmbr_shadow/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
- ./people/plenum-session-voters (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./votes/view_vote_rslts_hdr_approved
Missing dependency: Couldn't open datapackage data/votes/view_vote_rslts_hdr_approved/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./plenum/kns_plenumsession
Missing dependency: Couldn't open datapackage data/plenum/kns_plenumsession/datapackage.json
- ./people/plenum-session-voters-stats (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./plenum/kns_plenumsession
Missing dependency: Couldn't open datapackage data/plenum/kns_plenumsession/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/plenum-session-voters
Missing dependency: Couldn't open datapackage data/people/plenum/session_voters/datapackage.json
- ./people/mk-voted-against-majority (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./votes/view_vote_rslts_hdr_approved
Missing dependency: Couldn't open datapackage data/votes/view_vote_rslts_hdr_approved/datapackage.json
- ./people/mk-party-discipline-stats (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./votes/view_vote_rslts_hdr_approved
Missing dependency: Couldn't open datapackage data/votes/view_vote_rslts_hdr_approved/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/mk-voted-against-majority
Missing dependency: Couldn't open datapackage data/people/mk_voted_against_majority/datapackage.json
- ./people/mk-party-discipline-reports (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/mk-voted-against-majority
Missing dependency: Couldn't open datapackage data/people/mk_voted_against_majority/datapackage.json
- ./people/mk-party-discipline-reports-export (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./people/mk-party-discipline-reports
Missing dependency: Couldn't open datapackage data/people/mk_party_discipline_reports/datapackage.json
- ./people/attendance/plenum-votes (*)(E)
Dirty dependency: Cannot run until dependency is executed: ./knesset/kns_knessetdates
Missing dependency: Couldn't open datapackage data/knesset/kns_knessetdates/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./people/plenum-session-voters
Missing dependency: Couldn't open datapackage data/people/plenum/session_voters/datapackage.json
Dirty dependency: Cannot run until dependency is executed: ./members/mk_individual
Missing dependency: Couldn't open datapackage data/members/mk_individual/datapackage.json
In [3]:
!{'dpp run --verbose ./committees/kns_committee'}
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 RUNNING ./committees/kns_committee
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Collecting dependencies
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Running async task
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Waiting for completion
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Async task starting
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Searching for existing caches
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 Building process chain:
[./committees/kns_committee:T_0] >>> INFO :- ..datapackage_pipelines_knesset.dataservice.processors.add_dataservice_collection_resource
[./committees/kns_committee:T_0] >>> INFO :- ..datapackage_pipelines_knesset.common.processors.throttle
[./committees/kns_committee:T_0] >>> INFO :- knesset.dump_to_path
[./committees/kns_committee:T_0] >>> INFO :- knesset.dump_to_sql
[./committees/kns_committee:T_0] >>> INFO :- (sink)
[./committees/kns_committee:T_0] >>> INFO :..datapackage_pipelines_knesset.dataservice.processors.add_dataservice_collection_resource: INFO :Loading dataservice resource from service api method KNS_Committee
[./committees/kns_committee:T_0] >>> INFO :..datapackage_pipelines_knesset.common.processors.throttle: INFO :throttling resource kns_committee: sleep_seconds=0.01
[./committees/kns_committee:T_0] >>> INFO :..datapackage_pipelines_knesset.dataservice.processors.add_dataservice_collection_resource: INFO :Processed 756 rows
[./committees/kns_committee:T_0] >>> INFO :..datapackage_pipelines_knesset.common.processors.throttle: INFO :Processed 756 rows
[./committees/kns_committee:T_0] >>> INFO :knesset.dump_to_path: INFO :Processed 756 rows
[./committees/kns_committee:T_0] >>> INFO :knesset.dump_to_sql: INFO :Processed 756 rows
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE /pipelines/committees/../datapackage_pipelines_knesset/common/processors/throttle.py
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE /pipelines/datapackage_pipelines_knesset/processors/dump_to_path.py
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE /usr/local/lib/python3.6/site-packages/datapackage_pipelines/manager/../lib/internal/sink.py
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE /pipelines/datapackage_pipelines_knesset/processors/dump_to_sql.py
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE /pipelines/committees/../datapackage_pipelines_knesset/dataservice/processors/add_dataservice_collection_resource.py
[./committees/kns_committee:T_0] >>> INFO :6d09bf66 DONE V ./committees/kns_committee {'.dpp': {'out-datapackage-url': '../data/committees/kns_committee/datapackage.json'}, 'bytes': None, 'count_of_rows': 756, 'dataset_name': '_', 'hash': 'a3ab317a3db01ca952f714ba1c9c938e'}
INFO :RESULTS:
INFO :SUCCESS: ./committees/kns_committee {'bytes': None, 'count_of_rows': 756, 'dataset_name': '_', 'hash': 'a3ab317a3db01ca952f714ba1c9c938e'}
In [4]:
KNS_COMMITTEE_DATAPACKAGE_PATH = './data/committees/kns_committee/datapackage.json'
Each package may contain multiple resources, let's see which resource names are available for the kns_committee package
In [5]:
from datapackage import Package
kns_committee_package = Package(KNS_COMMITTEE_DATAPACKAGE_PATH)
kns_committee_package.resource_names
Out[5]:
['kns_committee']
In [6]:
KNS_COMMITTEE_RESOURE_NAME = 'kns_committee'
Inspect the kns_committee resource descriptor which includes metadata and field descriptions
In [7]:
import yaml
print(yaml.dump(kns_committee_package.get_resource(KNS_COMMITTEE_RESOURE_NAME).descriptor,
allow_unicode=True, default_flow_style=False))
bytes: 181959
count_of_rows: 756
dialect:
delimiter: ','
doubleQuote: true
lineTerminator: "\r\n"
quoteChar: '"'
skipInitialSpace: false
encoding: utf-8
format: csv
hash: 491306a262000aff5f4154905f96b31f
name: kns_committee
path: kns_committee.csv
profile: data-resource
schema:
fields:
- description: קוד הוועדה
name: CommitteeID
type: integer
- description: שם הוועדה
name: Name
type: string
- description: קוד הקטגוריה של הוועדה
name: CategoryID
type: integer
- description: 'תיאור הקטגוריה של הוועדה בכל כנסת, כל הוועדות מוקמות מחדש. השדה
קטגוריה כולל את רשימת הקטגוריות הנושאיות שאליהן משויכות הוועדות. למשל הקטגוריה
של ועדת הפנים והגנת הסביבה היא "פנים" וכך היה גם כאשר שם הוועדה היה ועדת הפנים
ואיכות הסביבה. גם ועדות המשנה של כל ועדה משויכות לקטגוריה שלה. מדובר בשיוך נושאי
של הוועדות.
'
name: CategoryDesc
type: string
- description: מספר הכנסת
name: KnessetNum
type: integer
- description: קוד סוג הוועדה
name: CommitteeTypeID
type: integer
- description: 'תיאור סוג הוועדה (ראשית, מיוחדת, משנה, משותפת, הכנסת)
'
name: CommitteeTypeDesc
type: string
- description: כתובת הדוא"ל של הוועדה
name: Email
type: string
- description: תאריך התחלה
format: '%Y-%m-%d %H:%M:%S'
name: StartDate
type: datetime
- description: תאריך סיום
format: '%Y-%m-%d %H:%M:%S'
name: FinishDate
type: datetime
- description: קוד סוג משנה של הוועדה
name: AdditionalTypeID
type: integer
- description: תיאור סוג משנה של הוועדה (קבועה, מיוחדת, חקירה)
name: AdditionalTypeDesc
type: string
- description: קוד ועדת האם (רלוונטי רק לוועדת משנה)
name: ParentCommitteeID
type: integer
- description: תיאור ועדת האם
name: CommitteeParentName
type: string
- description: האם הוועדה פעילה?
name: IsCurrent
type: boolean
- description: תאריך עדכון אחרון
format: '%Y-%m-%d %H:%M:%S'
name: LastUpdatedDate
type: datetime
Print the first 5 row of data
In [8]:
for i, row in enumerate(kns_committee_package.get_resource(KNS_COMMITTEE_RESOURE_NAME).iter(keyed=True), 1):
if i > 5: continue
print(f'-- row {i} --')
print(yaml.dump(row, allow_unicode=True, default_flow_style=False))
-- row 1 --
AdditionalTypeDesc: קבועה
AdditionalTypeID: 991
CategoryDesc: ועדת הכנסת
CategoryID: 1
CommitteeID: 1
CommitteeParentName: null
CommitteeTypeDesc: ועדת הכנסת
CommitteeTypeID: 70
Email: vadatk@knesset.gov.il
FinishDate: null
IsCurrent: true
KnessetNum: 15
LastUpdatedDate: 2017-04-24 16:47:06
Name: הכנסת
ParentCommitteeID: null
StartDate: 1999-06-07 00:00:00
-- row 2 --
AdditionalTypeDesc: קבועה
AdditionalTypeID: 991
CategoryDesc: ועדת הכספים
CategoryID: 2
CommitteeID: 2
CommitteeParentName: null
CommitteeTypeDesc: ועדה ראשית
CommitteeTypeID: 71
Email: null
FinishDate: null
IsCurrent: true
KnessetNum: 15
LastUpdatedDate: 2015-03-20 12:02:57
Name: הכספים
ParentCommitteeID: null
StartDate: 1999-06-07 00:00:00
-- row 3 --
AdditionalTypeDesc: קבועה
AdditionalTypeID: 991
CategoryDesc: ועדת החוץ והביטחון
CategoryID: 4
CommitteeID: 3
CommitteeParentName: null
CommitteeTypeDesc: ועדה ראשית
CommitteeTypeID: 71
Email: null
FinishDate: null
IsCurrent: true
KnessetNum: 15
LastUpdatedDate: 2015-03-20 12:02:57
Name: החוץ והבטחון
ParentCommitteeID: null
StartDate: 1999-06-07 00:00:00
-- row 4 --
AdditionalTypeDesc: קבועה
AdditionalTypeID: 991
CategoryDesc: ועדת הכלכלה
CategoryID: 3
CommitteeID: 4
CommitteeParentName: null
CommitteeTypeDesc: ועדה ראשית
CommitteeTypeID: 71
Email: null
FinishDate: null
IsCurrent: true
KnessetNum: 15
LastUpdatedDate: 2015-03-20 12:02:57
Name: הכלכלה
ParentCommitteeID: null
StartDate: 1999-06-07 00:00:00
-- row 5 --
AdditionalTypeDesc: קבועה
AdditionalTypeID: 991
CategoryDesc: ועדת הפנים והגנת הסביבה
CategoryID: 5
CommitteeID: 5
CommitteeParentName: null
CommitteeTypeDesc: ועדה ראשית
CommitteeTypeID: 71
Email: null
FinishDate: null
IsCurrent: true
KnessetNum: 15
LastUpdatedDate: 2015-03-20 12:02:57
Name: הפנים ואיכות הסביבה
ParentCommitteeID: null
StartDate: 1999-06-07 00:00:00
Content source: hasadna/knesset-data-pipelines
Similar notebooks: