ARCCSSive

Searching what's on NCI

The documentation for the ARCCSS CMIP5 data searching tools can be found here.

To use this on the CWS Virtual Desktops, I first had to comment out the miniconda command in my .bashrc file so that it used the default system python. Then I just typed the following before launching the notebook:

vdi $ pip install --user ARCCSSive
vdi $ export CMIP5_DB=sqlite:////g/data1/ua6/unofficial-ESG-replica/tmp/tree/cmip5_raijin_latest.db



In [1]:

    
from ARCCSSive import CMIP5
import six
import glob

cmip5 = CMIP5.DB.connect()



In [2]:

    
experiment = 'historicalMisc'
variable = 'tauuo'
mip = 'Omon'
model = 'CCSM4'
ensemble = 'r6i1p10'

outputs = cmip5.outputs(experiment = experiment,
                        variable = variable,
                        mip = mip,
                        model = model,
                        ensemble = ensemble)

path = '/g/data/ua6/DRSv2/CMIP5/%s/%s/mon/ocean/%s/%s/latest/*' %(model, experiment, ensemble, variable)
print(glob.glob(path))

for o in outputs:
    var = o.variable
    for v in o.versions:
        print(v.path)

for f in outputs.first().filenames():
    six.print_(f)









    



['/g/data/ua6/DRSv2/CMIP5/CCSM4/historicalMisc/mon/ocean/r6i1p10/tauuo/latest/tauuo_Omon_CCSM4_historicalMisc_r6i1p10_185001-200512.nc']
/g/data/ua6/unofficial-ESG-replica/tmp/tree/esgf2.dkrz.de/thredds/fileServer/cmip5/output1/NCAR/CCSM4/historicalMisc/mon/ocean/Omon/r6i1p10/v20121128/tauuo
tauuo_Omon_CCSM4_historicalMisc_r6i1p10_185001-200512.nc

The local install of ARCCSSive can get dated, so you need occasionally need to remove the source files and re-install. e.g:

vdi $ rm -r /home/599/dbi599/.local/lib/python2.7/site-packages/ARCCSSive/
vdi $ rm -r /home/599/dbi599/.local/lib/python2.7/site-packages/ARCCSSive-0.1.3.dev88+gba1abb-py2.7.egg-info/ 
vdi $ pip install --user ARCCSSive 
vdi $ pip freeze | grep -i arc 
ARCCSSive==0.2.2



In [28]:

    
for o in outputs:
    #print(o.drstree_path)
    print(o.ensemble)
    print(o.experiment)
    #print(o.filenames)
    #print(o.id)
    #print(o.latest)
    #print(o.metadata)
    print(o.mip)
    print(o.model)
    print(o.realm)
    for version in o.versions:
        #print(version.checked_on)
        #print(version.dataset_id)
        #print(version.drstree_path)
        print(version.files[0])
        #print(version.glob)
        #print(version.id)
        #print(version.instance_id)
        print(version.is_latest)
        print(version.metadata)
        print(version.path)
        print(version.to_update)
        print(version.tracking_ids())
        print(dir(version.variable))
        print(version.version)
        print(version.warnings)
        print(version.variable.variable)









    



r6i1p10
historicalMisc
Omon
CCSM4
ocean
[<ARCCSSive.CMIP5.Model.VersionFile object at 0x32e50d0>]
None
MetaData(bind=None)
/g/data/ua6/unofficial-ESG-replica/tmp/tree/esgf2.dkrz.de/thredds/fileServer/cmip5/output1/NCAR/CCSM4/historicalMisc/mon/ocean/Omon/r6i1p10/v20121128/tauuo
None
[u'bfabed9b-257c-47f5-8326-a44ca52616db']
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__mapper__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__table__', '__table_args__', '__tablename__', '__weakref__', '_decl_class_registry', '_sa_class_manager', '_sa_instance_state', 'drstree_path', 'ensemble', 'experiment', 'filenames', 'id', 'latest', 'metadata', 'mip', 'model', 'realm', 'variable', 'versions']
v20121128
[]
tauuo

Finding out what's on ESGF but not NCI

There's a script within the ARCCSS repo called compare_ESGF.py:

$ git clone https://github.com/coecms/ARCCSSive.git
$ cd ARCCSSive/examples
$ module load python
$ export CMIP5_DB=sqlite:////g/data1/ua6/unofficial-ESG-replica/tmp/tree/cmip5_raijin_latest.db
$ python compare_ESGF.py -v thetao -e historicalMisc -t Omon



In [21]:

    
outputs









    Out[21]:





<sqlalchemy.orm.query.Query at 0x32d33d0>



In [ ]: