notebook.community

Edit and run

Origin and goals of commonlib

  1. commonlib was written as a basis for a future lite version of the engine, with no dependency on Postgresql, Django and Celery
  2. the idea is to move in commonlib everything which does not depend on Postgresql, Django and Celery
  3. the engine must become small
  4. it must be possible to write hazard/risk calculators entirely in commonlib
  5. sooner or later the engine QA test will be moved in commonlib
  6. hazardlib, risklib, commonlib might become a single repository (oq-lite)

Use of commonlib: reading the engine input files





In [ ]:



    


import os; print os.getcwd()
print [f for f in os.listdir('.') if f.endswith(('.py', '.ini', '.xml'))]



    











In [ ]:



    


print open('job.ini').read()



    











In [ ]:



    


from openquake.baselib.general import import_all; import_all('my_calculators')



    











In [ ]:



    


from openquake.commonlib import readinput



    











In [ ]:



    


oqparam = readinput.get_oqparam('job.ini'); oqparam



    











In [ ]:



    


for name, value in oqparam:
    print name, '=', repr(value)



    











In [ ]:



    


for coords in oqparam.sites: print coords



    











In [ ]:



    


mesh = readinput.get_mesh(oqparam)
for site in mesh: print site



    











In [ ]:



    


[gsim] = readinput.get_gsims(oqparam); gsim



    











In [ ]:



    


print open('rupture_model.xml').read()



    











In [ ]:



    


rupture = readinput.get_rupture(oqparam); rupture



    











In [ ]:



    


rupture.mag, rupture.rake, rupture.surface



    











In [ ]:



    


imts = readinput.get_imts(oqparam); imts



    











In [ ]:



    


type(imts[0])



    











In [ ]:



    


sitecol = readinput.get_site_collection(oqparam); sitecol



    











In [ ]:



    


list(sitecol)

This is enough for performing simple hazard calculations





In [ ]:



    


from openquake.hazardlib.calc.gmf import ground_motion_fields



    











In [ ]:



    


ground_motion_fields?



    











In [ ]:



    


ground_motion_fields(rupture, sitecol, imts, gsim, oqparam.truncation_level, 
                     oqparam.number_of_ground_motion_fields,
                     correlation_model=readinput.get_correl_model(oqparam),
                     seed=oqparam.random_seed)

Reading risk input files





In [ ]:



    


oqp = readinput.get_oqparam('job_damage.ini')
for name, value in oqp:
    print name, value



    











In [ ]:



    


exposure = readinput.get_exposure(oqp); exposure



    











In [ ]:



    


a1 = exposure.assets[0]



    











In [ ]:



    


a1.number



    











In [ ]:



    


a1.location



    











In [ ]:



    


help(readinput.get_exposure)



    











In [ ]:



    


sitecol, assets_by_site = readinput.get_sitecol_assets(oqp, exposure)
sitecol, assets_by_site



    











In [ ]:



    


risk_model = readinput.get_risk_model(oqp); risk_model



    











In [ ]:



    


risk_model.keys()



    











In [ ]:



    


workflow = risk_model['PGA', 'RM']; workflow



    











In [ ]:



    


workflow.risk_functions



    











In [ ]:



    


print open('fragility_model.xml').read()



    











In [ ]:



    


workflow.risk_functions['damage'][0], workflow.risk_functions['damage'][1]



    











In [ ]:



    


risk_model.damage_states



    











In [ ]:



    


?workflow



    











In [ ]:



    


oqr = readinput.get_oqparam('job_risk.ini')
rm = readinput.get_risk_model(oqr)
rm.keys()



    











In [ ]:



    


rm['SA(0.2)', 'RC'].risk_functions

Reading sources





In [ ]:



    


from openquake.commonlib import readinput
usparam = readinput.get_oqparam('job_usa.ini')
sitecol = readinput.get_site_collection(usparam)



    











In [ ]:



    


source_models = list(readinput.get_composite_source_model(usparam, sitecol))  # slow
source_models



    











In [ ]:



    


trt_model = source_models[0].trt_models[0]
trt_model.min_mag, trt_model.max_mag, trt_model.num_ruptures



    











In [ ]:



    


trt_model.sources[:3], trt_model.sources[-3:]



    











In [ ]:



    


list(trt_model.sources[0].iter_ruptures())



    











In [ ]:



    


src = trt_model.sources[0]
src.mfd

Writing a HelloWorld calculator





In [ ]:



    


from openquake.calculators import base



    











In [ ]:



    


HelloCalculator = base.calculators['hello']; HelloCalculator



    











In [ ]:



    


print open('hello.ini').read()
oqp = readinput.get_oqparam('hello.ini')



    











In [ ]:



    


from openquake.baselib.performance import PerformanceMonitor
calc = HelloCalculator(oqp, PerformanceMonitor('hello'))



    











In [ ]:



    


calc.run()

In the export_dir there will be a file performance.csv with some interesting information about time spent and memory allocated

Testing the calculator





In [ ]:



    


from hello_test import HelloTestCase
HelloTestCase??

Parallel execution with apply_reduce

Example: building a frequency hystogram





In [ ]:



    


from collections import Counter



    











In [ ]:



    


Counter('pippolippo')



    











In [ ]:



    


def frequencyhisto(fnames):
    "Compute the character frequency hystogram of a set of files"
    c = Counter()
    for fname in fnames:
        c += Counter(open(fname).read().upper())
    return c



    











In [ ]:



    


from openquake.commonlib import nrml_examples



    











In [ ]:



    


DIR = nrml_examples.__path__[0]



    











In [ ]:



    


DIR



    











In [ ]:



    


import os; xmlfiles = [os.path.join(DIR, f) for f in os.listdir(DIR) if f.endswith('.xml')]; xmlfiles



    











In [ ]:



    


len(xmlfiles)



    











In [ ]:



    


apply(frequencyhisto, (xmlfiles,))



    











In [ ]:



    


from openquake.commonlib.parallel import apply_reduce



    











In [ ]:



    


apply_reduce(frequencyhisto, (xmlfiles,), acc=Counter(), concurrent_tasks=4)



    











In [ ]:



    


[len(chunk) for chunk in apply_reduce._chunks]



    











In [ ]:



    


apply_reduce(frequencyhisto, (xmlfiles,), acc=Counter(), concurrent_tasks=4,
             weight=os.path.getsize)



    











In [ ]:



    


[chunk.weight for chunk in apply_reduce._chunks]



    











In [ ]:



    


[len(chunk) for chunk in apply_reduce._chunks]



    











In [ ]:



    


apply_reduce._chunks[0]



    











In [ ]:



    


from openquake.baselib.general import split_in_blocks



    











In [ ]:



    


split_in_blocks??

Accumulating dictionaries





In [1]:



    


from openquake.baselib.general import AccumDict
AccumDict??



    











In [2]:



    


acc = AccumDict()
acc



    


















    
Out[2]:








{}

















In [3]:



    


acc2 = acc + {'a': 1}



    











In [4]:



    


acc3 = acc2 + {'a': 1}



    











In [5]:



    


acc3



    


















    
Out[5]:








{'a': 2}

















In [6]:



    


acc4 = acc3 + {'b': 0}
acc4



    


















    
Out[6]:








{'a': 2, 'b': 0}

Sequential scenario hazard calculator





In [7]:



    


from my_calculators import scenario
scenario??

Parallel scenario hazard calculator





In [9]:



    


from openquake.calculators.scenario import ScenarioCalculator



    











In [10]:



    


ScenarioCalculator??

Caveats and notes

  1. is the parallel calculator faster than the sequential one?
  2. is the export faster than the calculation?
  3. how much the results depend on the seed?
  4. is the memory occupation an issue?
  5. do we really need a scenario hazard calculator?

Content source: g-weatherill/oq-risklib

Similar notebooks:

notebook.community | gallery | about