Pymrio includes several functions for data reading and storing. This section presents the methods to use for saving and loading data already in a pymrio compatible format. For parsing raw MRIO data see the different tutorials for working with available MRIO databases.
Here, we use the included small test MRIO system to highlight the different function. The same functions are available for any MRIO loaded into pymrio. Expect, however, significantly decreased performance due to the size of real MRIO system.
In [1]:
import pymrio
import os
io = pymrio.load_test().calc_all()
To save the full system, use:
In [2]:
save_folder_full = '/tmp/testmrio/full'
io.save_all(path=save_folder_full)
Out[2]:
To read again from that folder do:
In [3]:
io_read = pymrio.load_all(path=save_folder_full)
The fileio activities are stored in the included meta data history field:
In [4]:
io_read.meta
Out[4]:
Internally, pymrio stores data in csv format, with the 'economic core' data in the root and each satellite account in a subfolder. Metadata as file as a file describing the data format ('file_parameters.json') are included in each folder.
In [5]:
import os
os.listdir(save_folder_full)
Out[5]:
The file format for storing the MRIO data can be switched to a binary pickle format with:
In [6]:
save_folder_bin = '/tmp/testmrio/binary'
io.save_all(path=save_folder_bin, table_format='pkl')
os.listdir(save_folder_bin)
Out[6]:
This can be used to reduce the storage space required on the disk for large MRIO databases.
To archive a MRIO system after saving use pymrio.archive:
In [7]:
mrio_arc = '/tmp/testmrio/archive.zip'
# Remove a potentially existing archive from before
try:
os.remove(mrio_arc)
except FileNotFoundError:
pass
pymrio.archive(source=save_folder_full, archive=mrio_arc)
Data can be read directly from such an archive by:
In [8]:
tt = pymrio.load_all(mrio_arc)
Currently data can not be saved directly into a zip archive. It is, however, possible to remove the source files after archiving:
In [9]:
tmp_save = '/tmp/testmrio/tmp'
# Remove a potentially existing archive from before
try:
os.remove(mrio_arc)
except FileNotFoundError:
pass
io.save_all(tmp_save)
print("Directories before archiving: {}".format(os.listdir('/tmp/testmrio')))
pymrio.archive(source=tmp_save, archive=mrio_arc, remove_source=True)
print("Directories after archiving: {}".format(os.listdir('/tmp/testmrio')))
Several MRIO databases can be stored in the same archive:
In [10]:
# Remove a potentially existing archive from before
try:
os.remove(mrio_arc)
except FileNotFoundError:
pass
tmp_save = '/tmp/testmrio/tmp'
io.save_all(tmp_save)
pymrio.archive(source=tmp_save, archive=mrio_arc, path_in_arc='version1/', remove_source=True)
io2 = io.copy()
del io2.emissions
io2.save_all(tmp_save)
pymrio.archive(source=tmp_save, archive=mrio_arc, path_in_arc='version2/', remove_source=True)
When loading from an archive which includes multiple MRIO databases, specify one with the parameter 'path_in_arc':
In [11]:
io1_load = pymrio.load_all(mrio_arc, path_in_arc='version1/')
io2_load = pymrio.load_all(mrio_arc, path_in_arc='version2/')
print("Extensions of the loaded io1 {ver1} and of io2: {ver2}".format(
ver1=list(io1_load.get_extensions()),
ver2=list(io2_load.get_extensions())))
The pymrio.load function can be used directly to only a specific satellite account of a MRIO database from a zip archive:
In [12]:
emissions = pymrio.load(mrio_arc, path_in_arc='version1/emissions')
print(emissions)
The archive function is a wrapper around python.zipfile module. There are, however, some differences to the defaults choosen in the original:
In contrast to zipfile.write, pymrio.archive raises an error if the data (path + filename) are identical in the zip archive. Background: the zip standard allows that files with the same name and path are stored side by side in a zip file. This becomes an issue when unpacking this files as they overwrite each other upon extraction.
The standard for the parameter 'compression' is set to ZIP_DEFLATED This is different from the zipfile default (ZIP_STORED) which would not give any compression. See the zipfile docs for further information. Depending on the value given for the parameter 'compression' additional modules might be necessary (e.g. zlib for ZIP_DEFLATED). Futher information on this can also be found in the zipfile python docs.
Each extension of the MRIO system can be stored separetly with:
In [13]:
save_folder_em= '/tmp/testmrio/emissions'
In [14]:
io.emissions.save(path=save_folder_em)
Out[14]:
This can then be loaded again as separate satellite account:
In [15]:
emissions = pymrio.load(save_folder_em)
In [16]:
emissions
Out[16]:
In [17]:
emissions.D_cba
Out[17]:
As all data in pymrio is stored as pandas DataFrame, the full pandas stack for exporting tables is available. For example, to export a table as excel sheet use:
In [18]:
io.emissions.D_cba.to_excel('/tmp/testmrio/emission_footprints.xlsx')
For further information see the pandas documentation on import/export.