Origin and goals of commonlib

  1. commonlib was written as a basis for a future lite version of the engine, with no dependency on Postgresql, Django and Celery
  2. the idea is to move in commonlib everything which does not depend on Postgresql, Django and Celery
  3. the engine must become small
  4. it must be possible to write hazard/risk calculators entirely in commonlib
  5. sooner or later the engine QA test will be moved in commonlib
  6. hazardlib, risklib, commonlib might become a single repository (oq-lite)

Use of commonlib: reading the engine input files


In [179]:
import os; print os.getcwd()
print [f for f in os.listdir('.') if f.endswith(('.py', '.ini', '.xml'))]


/home/ubuntu/oq-risklib/docs
['fault_rupture.xml', 'NSHMP2008US_micro.xml', 'test.py', 'job_usa.ini', 'gmpe_logic_tree.xml', 'job_hc_from_csv.ini', 'source_model_logic_tree.xml', 'job_gmvs_from_csv.ini', 'job_damage.ini', 'job.ini', 'vulnerability_model.xml', 'riskinput.py', 'conf.py', 'job_risk.ini', 'rupture_model.xml', 'fragility_model.xml', 'expected.xml', 'hello_test.py', 'hello.ini', 'exposure_model.xml']

In [180]:
print open('job.ini').read()


[general]
description = Scenario QA Test, Case 1
calculation_mode = my_scenario
export_dir = /tmp
random_seed = 3
sites = 0.0 0.0, 0.0 0.1, 0.0 0.2
rupture_mesh_spacing = 1.0
reference_vs30_type = measured
reference_vs30_value = 800.0
reference_depth_to_2pt5km_per_sec = 2.5
reference_depth_to_1pt0km_per_sec = 100.0
rupture_model_file = rupture_model.xml
intensity_measure_types = PGA
truncation_level = 1.0
maximum_distance = 200
gsim = BooreAtkinson2008
number_of_ground_motion_fields = 5


In [181]:
from openquake.baselib.general import import_all; import_all('my_calculators')


Out[181]:
set()

In [182]:
from openquake.commonlib import readinput

In [183]:
oqparam = readinput.get_oqparam('job.ini'); oqparam


Out[183]:
<OqParam base_path=/home/ubuntu/oq-risklib/docs, calculation_mode=my_scenario, complex_fault_mesh_spacing=1.0, description=Scenario QA Test, Case 1, export_dir=/tmp, gsim=BooreAtkinson2008, hazard_imtls={'PGA': None}, inputs={'rupture_model': '/home/ubuntu/oq-risklib/docs/rupture_model.xml', 'job_ini': 'job.ini'}, maximum_distance=200.0, number_of_ground_motion_fields=5, random_seed=3, reference_depth_to_1pt0km_per_sec=100.0, reference_depth_to_2pt5km_per_sec=2.5, reference_vs30_type=measured, reference_vs30_value=800.0, rupture_mesh_spacing=1.0, sites=[(0.0, 0.0), (0.0, 0.1), (0.0, 0.2)], truncation_level=1.0>

In [184]:
for name, value in oqparam:
    print name, '=', repr(value)


base_path = '/home/ubuntu/oq-risklib/docs'
calculation_mode = 'my_scenario'
complex_fault_mesh_spacing = 1.0
description = 'Scenario QA Test, Case 1'
export_dir = '/tmp'
gsim = 'BooreAtkinson2008'
hazard_imtls = {'PGA': None}
inputs = {'rupture_model': '/home/ubuntu/oq-risklib/docs/rupture_model.xml', 'job_ini': 'job.ini'}
maximum_distance = 200.0
number_of_ground_motion_fields = 5
random_seed = 3
reference_depth_to_1pt0km_per_sec = 100.0
reference_depth_to_2pt5km_per_sec = 2.5
reference_vs30_type = 'measured'
reference_vs30_value = 800.0
rupture_mesh_spacing = 1.0
sites = [(0.0, 0.0), (0.0, 0.1), (0.0, 0.2)]
truncation_level = 1.0

In [185]:
for coords in oqparam.sites: print coords


(0.0, 0.0)
(0.0, 0.1)
(0.0, 0.2)

In [186]:
mesh = readinput.get_mesh(oqparam)
for site in mesh: print site


<Latitude=0.000000, Longitude=0.000000, Depth=0.0000>
<Latitude=0.100000, Longitude=0.000000, Depth=0.0000>
<Latitude=0.200000, Longitude=0.000000, Depth=0.0000>

In [187]:
[gsim] = readinput.get_gsims(oqparam); gsim


Out[187]:
<openquake.hazardlib.gsim.boore_atkinson_2008.BooreAtkinson2008 at 0x4759550>

In [188]:
print open('rupture_model.xml').read()


<?xml version='1.0' encoding='utf-8'?>
<nrml xmlns:gml="http://www.opengis.net/gml"
      xmlns="http://openquake.org/xmlns/nrml/0.4">
    <simpleFaultRupture>
        <magnitude>7.0</magnitude>
        <rake>90.0</rake>
        <hypocenter lat="0.0" lon="0.0" depth="10.0"/>
        <simpleFaultGeometry>
            <gml:LineString>
                <gml:posList>
                    28 80.3
                    28  80.6
                </gml:posList>
            </gml:LineString>
            <dip>90.0</dip>
            <upperSeismoDepth>2.0</upperSeismoDepth>
            <lowerSeismoDepth>20.0</lowerSeismoDepth>
        </simpleFaultGeometry>
    </simpleFaultRupture>
</nrml>


In [189]:
rupture = readinput.get_rupture(oqparam); rupture


Out[189]:
<openquake.hazardlib.source.rupture.Rupture at 0x4f139b0>

In [190]:
rupture.mag, rupture.rake, rupture.surface


Out[190]:
(7.0,
 90.0,
 <openquake.hazardlib.geo.surface.simple_fault.SimpleFaultSurface at 0x47640d0>)

In [191]:
imts = readinput.get_imts(oqparam); imts


Out[191]:
[PGA()]

In [192]:
type(imts[0])


Out[192]:
openquake.hazardlib.imt.PGA

In [193]:
sitecol = readinput.get_site_collection(oqparam); sitecol


Out[193]:
<SiteCollection with 3 sites>

In [194]:
list(sitecol)


Out[194]:
[<Location=<Latitude=0.000000, Longitude=0.000000, Depth=0.0000>, Vs30=800.0000, Vs30Measured=True, Depth1.0km=100.0000, Depth2.5km=2.5000>,
 <Location=<Latitude=0.100000, Longitude=0.000000, Depth=0.0000>, Vs30=800.0000, Vs30Measured=True, Depth1.0km=100.0000, Depth2.5km=2.5000>,
 <Location=<Latitude=0.200000, Longitude=0.000000, Depth=0.0000>, Vs30=800.0000, Vs30Measured=True, Depth1.0km=100.0000, Depth2.5km=2.5000>]

This is enough for performing simple hazard calculations


In [195]:
from openquake.hazardlib.calc.gmf import ground_motion_fields

In [196]:
ground_motion_fields?

In [197]:
ground_motion_fields(rupture, sitecol, imts, gsim, oqparam.truncation_level, 
                     oqparam.number_of_ground_motion_fields,
                     correlation_model=readinput.get_correl_model(oqparam),
                     seed=oqparam.random_seed)


Out[197]:
OrderedDict([(PGA(), array([[  1.29778834e-47,   1.12319707e-47,   1.01915100e-47,
          1.07825099e-47,   1.65947553e-47],
       [  2.04784726e-47,   7.51203345e-48,   1.06991019e-47,
          7.86882106e-48,   1.24054768e-47],
       [  1.01174143e-47,   1.16454566e-47,   1.80072257e-47,
          1.13539698e-47,   1.73028861e-47]]))])

Reading risk input files


In [198]:
oqp = readinput.get_oqparam('job_damage.ini')
for name, value in oqp:
    print name, value


base_path /home/ubuntu/oq-risklib/docs
calculation_mode scenario_damage
complex_fault_mesh_spacing 10.0
continuous_fragility_discretization 20
description Scenario Damage Example
export_dir /tmp
ground_motion_correlation_model None
ground_motion_correlation_params {}
gsim ChiouYoungs2008
inputs {'rupture_model': '/home/ubuntu/oq-risklib/docs/fault_rupture.xml', 'fragility': '/home/ubuntu/oq-risklib/docs/fragility_model.xml', 'job_ini': 'job_damage.ini', 'exposure': '/home/ubuntu/oq-risklib/docs/exposure_model.xml'}
maximum_distance 300.0
number_of_ground_motion_fields 5
random_seed 3
reference_depth_to_1pt0km_per_sec 100.0
reference_depth_to_2pt5km_per_sec 5.0
reference_vs30_type measured
reference_vs30_value 760.0
region_constraint POLYGON((78.0 31.5, 89.5 31.5, 89.5 25.5, 78.0 25.5, 78.0 31.5))
risk_imtls {'SA(0.5)': [0.02, 0.09789473684210527, 0.17578947368421052, 0.25368421052631579, 0.33157894736842108, 0.40947368421052632, 0.48736842105263162, 0.56526315789473691, 0.64315789473684215, 0.72105263157894739, 0.79894736842105263, 0.87684210526315798, 0.95473684210526322, 1.0326315789473686, 1.1105263157894738, 1.188421052631579, 1.2663157894736843, 1.3442105263157895, 1.4221052631578948, 1.5], 'SA(0.2)': [0.02, 0.09789473684210527, 0.17578947368421052, 0.25368421052631579, 0.33157894736842108, 0.40947368421052632, 0.48736842105263162, 0.56526315789473691, 0.64315789473684215, 0.72105263157894739, 0.79894736842105263, 0.87684210526315798, 0.95473684210526322, 1.0326315789473686, 1.1105263157894738, 1.188421052631579, 1.2663157894736843, 1.3442105263157895, 1.4221052631578948, 1.5], 'PGA': [0.02, 0.09789473684210527, 0.17578947368421052, 0.25368421052631579, 0.33157894736842108, 0.40947368421052632, 0.48736842105263162, 0.56526315789473691, 0.64315789473684215, 0.72105263157894739, 0.79894736842105263, 0.87684210526315798, 0.95473684210526322, 1.0326315789473686, 1.1105263157894738, 1.188421052631579, 1.2663157894736843, 1.3442105263157895, 1.4221052631578948, 1.5]}
rupture_mesh_spacing 10.0
truncation_level 3.0

In [199]:
exposure = readinput.get_exposure(oqp); exposure


Out[199]:
Exposure(id='ep', category='buildings', description='Exposure model for buildings', cost_types=[{'type': 'per_asset', 'name': 'structural', 'unit': 'USD'}], insurance_limit_is_absolute=None, deductible_is_absolute=None, assets=[<Asset a1>, <Asset a2>, <Asset a3>], taxonomies=set(['RM', 'W', 'RC']))

In [200]:
a1 = exposure.assets[0]

In [201]:
a1.number


Out[201]:
3000.0

In [202]:
a1.location


Out[202]:
(81.2985, 29.1098)

In [203]:
help(readinput.get_exposure)


Help on function get_exposure in module openquake.commonlib.readinput:

get_exposure(oqparam)
    Read the full exposure in memory and build a list of
    :class:`openquake.risklib.workflows.Asset` instances.
    If you don't want to keep everything in memory, use
    get_exposure_lazy instead (for experts only).
    
    :param oqparam:
        an :class:`openquake.commonlib.oqvalidation.OqParam` instance
    :returns:
        an :class:`Exposure` instance


In [204]:
sitecol, assets_by_site = readinput.get_sitecol_assets(oqp, exposure)
sitecol, assets_by_site


Out[204]:
(<SiteCollection with 3 sites>, array([[<Asset a1>],
        [<Asset a2>],
        [<Asset a3>]], dtype=object))

In [205]:
risk_model = readinput.get_risk_model(oqp); risk_model


Out[205]:
<openquake.risklib.riskinput.RiskModel at 0x4759850>

In [206]:
risk_model.keys()


Out[206]:
[('PGA', 'RM'), ('SA(0.2)', 'RC'), ('SA(0.5)', 'W')]

In [207]:
workflow = risk_model['PGA', 'RM']; workflow


Out[207]:
<openquake.risklib.workflows.Damage at 0x4f2cbd0>

In [208]:
workflow.risk_functions


Out[208]:
{'damage': <FragilityFunctionList steps_per_interval=None, continuous_fragility_discretization=20, no_damage_limit=None, imt=PGA, imls=[0.02, 0.09789473684210527, 0.17578947368421052, 0.25368421052631579, 0.33157894736842108, 0.40947368421052632, 0.48736842105263162, 0.56526315789473691, 0.64315789473684215, 0.72105263157894739, 0.79894736842105263, 0.87684210526315798, 0.95473684210526322, 1.0326315789473686, 1.1105263157894738, 1.188421052631579, 1.2663157894736843, 1.3442105263157895, 1.4221052631578948, 1.5]>}

In [209]:
print open('fragility_model.xml').read()


<?xml version='1.0' encoding='utf-8'?>
<nrml xmlns="http://openquake.org/xmlns/nrml/0.4">
  <fragilityModel format="continuous">
    <description>Fragility model for buildings</description>
    <limitStates>
      LS1
      LS2
    </limitStates>
    <ffs type="lognormal">
      <taxonomy>RM</taxonomy>
      <IML IMT="PGA" imlUnit="g" minIML="0.02" maxIML="1.5"/>
      <ffc ls="LS1">
	<params mean="0.15" stddev="0.10" />
      </ffc>
      <ffc ls="LS2">
	<params mean="0.30" stddev="0.16" />
      </ffc>
    </ffs>
    <ffs type="lognormal">
      <taxonomy>RC</taxonomy>
      <IML IMT="SA(0.2)" imlUnit="g" minIML="0.02" maxIML="1.5"/>
      <ffc ls="LS1">
	<params mean="0.25" stddev="0.08" />
      </ffc>
      <ffc ls="LS2">
	<params mean="0.55" stddev="0.16" />
      </ffc>
    </ffs>
    <ffs type="lognormal">
      <taxonomy>W</taxonomy>
      <IML IMT="SA(0.5)" imlUnit="g" minIML="0.02" maxIML="1.5"/>
      <ffc ls="LS1">
	<params mean="0.35" stddev="0.12" />
      </ffc>
      <ffc ls="LS2">
	<params mean="0.70" stddev="0.20" />
      </ffc>
    </ffs>
  </fragilityModel>
</nrml>


In [210]:
workflow.risk_functions['damage'][0], workflow.risk_functions['damage'][1]


Out[210]:
(<FragilityFunctionContinuous(LS1, 0.15, 0.1)>,
 <FragilityFunctionContinuous(LS2, 0.3, 0.16)>)

In [211]:
risk_model.damage_states


Out[211]:
['no_damage', 'LS1', 'LS2']

In [212]:
?workflow

In [213]:
oqr = readinput.get_oqparam('job_risk.ini')
rm = readinput.get_risk_model(oqr)
rm.keys()


Out[213]:
[('PGA', 'RM'), ('SA(0.2)', 'RC'), ('SA(0.5)', 'W')]

In [214]:
rm['SA(0.2)', 'RC'].risk_functions


Out[214]:
{'structural': <VulnerabilityFunction(SA(0.2))>}

Reading sources


In [215]:
from openquake.commonlib import readinput
usparam = readinput.get_oqparam('job_usa.ini')
sitecol = readinput.get_site_collection(usparam)

In [216]:
source_models = list(readinput.get_composite_source_model(usparam, sitecol))  # slow
source_models


Out[216]:
[SourceModel(name='NSHMP2008US_micro.xml', weight=Decimal('1.00'), path=('b1',), trt_models=[<TrtModel #0 Active Shallow Crust, 5612 source(s)>], gsim_lt=<openquake.commonlib.logictree.GsimLogicTree object at 0x4b15890>, ordinal=0)]

In [217]:
trt_model = source_models[0].trt_models[0]
trt_model.min_mag, trt_model.max_mag, trt_model.num_ruptures


Out[217]:
(5.369, 8.42, 9310)

In [218]:
trt_model.sources[:3], trt_model.sources[-3:]


Out[218]:
([<CharacteristicFaultSource 1>,
  <CharacteristicFaultSource 100>,
  <CharacteristicFaultSource 1000>],
 [<CharacteristicFaultSource 997>,
  <CharacteristicFaultSource 998>,
  <CharacteristicFaultSource 999>])

In [219]:
list(trt_model.sources[0].iter_ruptures())


Out[219]:
[<openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250100>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250260>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250310>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x62503c0>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250470>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250520>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x62505d0>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250680>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250730>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x62507e0>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250890>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250940>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x62509f0>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250aa0>,
 <openquake.hazardlib.source.rupture.ParametricProbabilisticRupture at 0x6250b50>]

In [220]:
src = trt_model.sources[0]
src.mfd


Out[220]:
<openquake.hazardlib.mfd.evenly_discretized.EvenlyDiscretizedMFD at 0x4f023c0>

Reading CSV files


In [221]:
o = readinput.get_oqparam('job_gmvs_from_csv.ini')
readinput.get_sitecol_gmfs(o)


Out[221]:
(<SiteCollection with 3 sites>,
 {'PGA': array([[ 0.489966,  0.455252,  0.393138],
         [ 0.207681,  0.240424,  0.300524],
         [ 0.099697,  0.087133,  0.092085]]),
  'PGV': array([[ 41.4227 ,  26.7931 ,  33.1676 ],
         [ 17.4768 ,   9.054  ,  13.2879 ],
         [  8.19412,   8.41794,   6.5504 ]])})

In [222]:
o = readinput.get_oqparam('job_hc_from_csv.ini')
readinput.get_sitecol_hcurves(o)


Out[222]:
(<SiteCollection with 3 sites>,
 {'PGA': array([[ 0.4891846 ,  0.45525296,  0.39313824],
         [ 0.30783781,  0.24033782,  0.13052459],
         [ 0.9977148 ,  0.87679151,  0.20852236]]),
  'SA(0.2)': array([[ 0.414227  ,  0.19310681,  0.16763226],
         [ 0.17196265,  0.05421363,  0.02879748],
         [ 0.81992743,  0.61794658,  0.55046728]]),
  'SA(0.5)': array([[ 0.0414227 ,  0.01931068,  0.01676323],
         [ 0.0171965 ,  0.00542136,  0.00287974],
         [ 0.8199274 ,  0.61794658,  0.55046728]])})

Writing a HelloWorld calculator


In [223]:
from openquake.commonlib.calculators import base

In [224]:
HelloCalculator = base.calculators['hello']; HelloCalculator


Out[224]:
my_calculators.hello.HelloCalculator

In [225]:
print open('hello.ini').read()
oqp = readinput.get_oqparam('hello.ini')


[general]
calculation_mode = hello
maximum_distance = 200
intensity_measure_types = PGA
export_dir = /tmp


In [226]:
from openquake.commonlib.parallel import PerformanceMonitor
calc = HelloCalculator(oqp, PerformanceMonitor('hello'))

In [227]:
calc.run()


Out[227]:
{'hello': '/tmp/hello.txt'}

In the export_dir there will be a file performance.csv with some interesting information about time spent and memory allocated

Testing the calculator


In [228]:
from hello_test import HelloTestCase
HelloTestCase??

===================================================================================================================================

Parallel execution with apply_reduce

Example: building a frequency hystogram


In [229]:
from collections import Counter

In [230]:
Counter('pippolippo')


Out[230]:
Counter({'p': 5, 'i': 2, 'o': 2, 'l': 1})

In [231]:
def frequencyhisto(fnames):
    "Compute the character frequency hystogram of a set of files"
    c = Counter()
    for fname in fnames:
        c += Counter(open(fname).read().upper())
    return c

In [232]:
from openquake.commonlib import nrml_examples

In [233]:
DIR = nrml_examples.__path__[0]

In [234]:
DIR


Out[234]:
'/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples'

In [235]:
import os; xmlfiles = [os.path.join(DIR, f) for f in os.listdir(DIR) if f.endswith('.xml')]; xmlfiles


Out[235]:
['/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/logic-tree-gmpe.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/logic-tree-source-model.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/complex-fault-rupture.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-fractions-taxonomies.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-mean.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/uhs-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/collapse-map.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/ses.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/site_model.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/dmg-dist-total.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-multi.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/exposure-population.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/vulnerability-model-discrete.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-map-mean.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-sa.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/bcr-map.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-map-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-pga.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/gmf-event-based.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/fragm_c.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/disaggregation.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/bcr-map-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-map-scenario-risk.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-curves-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/exposure-portfolio.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/agg-loss-curve.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/uhs-mean.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/uhs.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/fragm_d.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-map.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/dmg-dist-per-taxonomy.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-curves.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/dmg-dist-per-asset.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-map-quantile.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/simple-fault-rupture.xml',
 '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-map.xml']

In [236]:
len(xmlfiles)


Out[236]:
37

In [237]:
apply(frequencyhisto, (xmlfiles,))


Out[237]:
Counter({' ': 12304, 'E': 2177, 'T': 1799, '"': 1726, 'S': 1545, 'N': 1410, 'L': 1389, '0': 1383, 'O': 1368, 'A': 1339, 'I': 1300, 'R': 1159, '\n': 1154, 'M': 1102, '<': 1052, '>': 1052, '.': 1049, '=': 940, '/': 861, 'G': 781, 'D': 761, 'P': 747, 'C': 699, '1': 586, 'U': 559, '2': 405, '-': 363, 'H': 353, 'V': 329, 'F': 312, ':': 286, '3': 275, 'B': 269, '4': 257, '5': 246, 'X': 219, 'Y': 211, '9': 181, '8': 177, '7': 167, '6': 167, 'W': 147, '_': 94, "'": 92, '?': 72, 'Z': 70, ',': 53, 'Q': 53, 'K': 52, '+': 41, '!': 40, '|': 30, '`': 16, ')': 6, '(': 6})

In [238]:
from openquake.commonlib.parallel import apply_reduce
import operator

In [239]:
apply_reduce(frequencyhisto, (xmlfiles,), agg=operator.add, acc=Counter(), concurrent_tasks=4)


Out[239]:
Counter({' ': 12304, 'E': 2177, 'T': 1799, '"': 1726, 'S': 1545, 'N': 1410, 'L': 1389, '0': 1383, 'O': 1368, 'A': 1339, 'I': 1300, 'R': 1159, '\n': 1154, 'M': 1102, '<': 1052, '>': 1052, '.': 1049, '=': 940, '/': 861, 'G': 781, 'D': 761, 'P': 747, 'C': 699, '1': 586, 'U': 559, '2': 405, '-': 363, 'H': 353, 'V': 329, 'F': 312, ':': 286, '3': 275, 'B': 269, '4': 257, '5': 246, 'X': 219, 'Y': 211, '9': 181, '8': 177, '7': 167, '6': 167, 'W': 147, '_': 94, "'": 92, '?': 72, 'Z': 70, ',': 53, 'Q': 53, 'K': 52, '+': 41, '!': 40, '|': 30, '`': 16, ')': 6, '(': 6})

In [240]:
[len(chunk) for chunk in apply_reduce._chunks]


Out[240]:
[10, 10, 10, 7]

In [241]:
apply_reduce(frequencyhisto, (xmlfiles,), agg=operator.add, acc=Counter(), concurrent_tasks=4,
             weight=os.path.getsize)


Out[241]:
Counter({' ': 12304, 'E': 2177, 'T': 1799, '"': 1726, 'S': 1545, 'N': 1410, 'L': 1389, '0': 1383, 'O': 1368, 'A': 1339, 'I': 1300, 'R': 1159, '\n': 1154, 'M': 1102, '<': 1052, '>': 1052, '.': 1049, '=': 940, '/': 861, 'G': 781, 'D': 761, 'P': 747, 'C': 699, '1': 586, 'U': 559, '2': 405, '-': 363, 'H': 353, 'V': 329, 'F': 312, ':': 286, '3': 275, 'B': 269, '4': 257, '5': 246, 'X': 219, 'Y': 211, '9': 181, '8': 177, '7': 167, '6': 167, 'W': 147, '_': 94, "'": 92, '?': 72, 'Z': 70, ',': 53, 'Q': 53, 'K': 52, '+': 41, '!': 40, '|': 30, '`': 16, ')': 6, '(': 6})

In [242]:
[chunk.weight for chunk in apply_reduce._chunks]


Out[242]:
[8742, 11239, 8761, 10360, 6129]

In [243]:
[len(chunk) for chunk in apply_reduce._chunks]


Out[243]:
[7, 10, 7, 8, 5]

In [244]:
apply_reduce._chunks[0]


Out[244]:
<WeightedSequence ['/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/logic-tree-gmpe.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/logic-tree-source-model.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/complex-fault-rupture.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/loss-fractions-taxonomies.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/hazard-curves-mean.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/uhs-quantile.xml', '/home/ubuntu/oq-risklib/openquake/commonlib/nrml_examples/collapse-map.xml'], weight=8742>

In [245]:
from openquake.baselib.general import split_in_blocks

In [246]:
split_in_blocks??

Accumulating dictionaries


In [247]:
from openquake.baselib.general import AccumDict
AccumDict??

In [248]:
acc = AccumDict()
acc


Out[248]:
{}

In [249]:
acc2 = acc + {'a': 1}

In [250]:
acc3 = acc2 + {'a': 1}

In [251]:
acc3


Out[251]:
{'a': 2}

In [252]:
acc4 = acc3 + {'b': 0}
acc4


Out[252]:
{'a': 2, 'b': 0}

Sequential scenario hazard calculator


In [253]:
from my_calculators import scenario
scenario??

Parallel scenario hazard calculator


In [254]:
from openquake.commonlib.calculators.hazard import ScenarioCalculator

In [255]:
ScenarioCalculator??

Caveats and notes

  1. is the parallel calculator faster than the sequential one?
  2. is the export faster than the calculation?
  3. how much the results depend on the seed?
  4. is the memory occupation an issue?
  5. do we really need a scenario hazard calculator?

In [256]:
from openquake.commonlib.parallel import PerformanceMonitor
PerformanceMonitor?

Towards a Scenario Damage Calculator


In [257]:
from openquake.commonlib import readinput
from openquake.commonlib.calculators.calc import calc_gmfs

In [258]:
o = readinput.get_oqparam('job_damage.ini')
exposure = readinput.get_exposure(o)
sitecol, assets_by_site = readinput.get_sitecol_assets(o, exposure)
risk_model = readinput.get_risk_model(o)
gmfs_by_imt = calc_gmfs(o, sitecol)

In [259]:
ri = risk_model.build_input('PGA', gmfs_by_imt['PGA'], assets_by_site)
ri


Out[259]:
<RiskInput IMT=PGA, taxonomy=RM, weight=1>

In [260]:
assets, hazards, epsilons = ri.get_all()
print assets
print hazards
print epsilons


{'RM': [<Asset a1>]}
{'RM': [array([ 0.06108985,  0.02149156,  0.09326082,  0.017891  ,  0.06110326])]}
{'RM': [None]}

In [261]:
from openquake.risklib.riskinput import RiskInput; RiskInput


Out[261]:
openquake.risklib.riskinput.RiskInput