ESGF replication request form

This form is intended to request data to be replicated from other ESGF nodes to be made locally available in the DKRZ CMIP data pool.

The specification of a requested data collection is based on the search facets describing the data collection. These facets correspond directly to the search categories you use to find data in one of the ESGF portals (e.g. https://esgf-data.dkrz.de/).

Specification of ESGF data to be replicated

To be able to automate the data replication process as much as possible we recommend the following steps, which are supported in this form. In case you have problems with this approach please contact us directly via mail (esgf-replication 'at' dkrz.de).

Step 1: define your data request based on the search facets you need to characterize the data collection in one of the ESGF portals.
Step 2: write down your facet selection choices in the specific format supported by the synda replication tool:
- The specification is based on so called selection files see examples for a set of examples
- specifiy the selection files characterizing your request in this part
Step 3: Test and check your selection file(s) with respect to correctnes
Step 4: Provide information on the context of your request
Step 5: Generate file lists associated to your replica request Check your selection file(s) with respect to data volume adressed
Step 6: Submit your replication request

General remarks:

We recommend to install the synda application at your lab in case you have recurring needs for data to be made available at DKRZ, this way you can prepare and verify your replication at your lab.
We recommend to split your request into a set of small well defined selection files instead of specifying one complex file characterizing your complete data needs

Please provide your last name and the password for this form

fill in your last name and evaluate the following cell ("SHIFT-ENTER"), you will then be asked for your password provided to you when requesting this form



In [ ]:

    
MY_LAST_NAME = "....."   # e.gl MY_LAST_NAME = "schulz" 

#------instantiate your replication form --------------------
from dkrz_forms import form_handler, form_widgets
form_info = form_widgets.check_pwd(MY_LAST_NAME)
sf = form_handler.init_form(form_info)
form = sf.sub.entity_out.report

Step 2: Edit and store your replica selection file(s)

Please provide the facet values charaterizing your data request. You can find the appropriate settings either

by using an ESGF portal and remembering your search facets or
by playing around with the cells below until your request is fully specified or
by installing the synda tool at your lab and using the tool directly at home - just copy the tested synda selection files into the slots below..

an example seclection file looks like:

 project="CMIP5"
 model="CNRM-CM5 CSIRO-Mk3-6-0"
 experiment="historical amip"
 ensemble="r1i1p1"
 variable[atmos][mon]="tasmin tas psl"
 variable[ocean][fx]="areacello sftof"
 variable[land][mon]="mrsos,nppRoot,nep"
 variable[seaIce][mon]="sic evap"
 variable[ocnBgchem][mon]="dissic fbddtalk"

You can store your request using the cells below by adding %%writefile seclection/myfilename.txt as a first line. Please select "myfilenamee" carefully to be able to remember later the dataset which this file charecterizes e.g. %%writefile erich_cmip5_atmos_vars_for_exp1.txt

store your selection files using the cells below

please provide the names of your selection files in the cell below and evaluate it ("SHIFT-ENTER")
for each file name an input field is generated to be filled with your data specification



In [ ]:

    
# provide the list of selection file names (.txt files)
# detailed, characterizing file names prefered .. 
# e.g. sel_file_list = ["cmip5_mpi-m_rcp_1","cmip5_smhi_rcp_0"]

sel_file_list = ["...","..."]      # strings in a list

#---- generation of input fields for your files
text_w = form_widgets.get_selection_files(sel_file_list)
form_widgets.gen_text_widgets(text_w)

Step3: Check your selection file(s)

using the cells below you can directly interact with the synda tool to check your selection files. The generic syntax is:

  - synda <command> -s ./selection/<your_selection_file>

see also the example synda cells in the Appendix of this form



In [ ]:

    
%%bash 
# The following command searches for data sets matching your request

synda search -s ./selection/your_selection_file_to_be_checked.txt 

# other helpfull commands
# synda check -s ./selection/...
# synda -h


# Final check always should be done with the following command
# - it shows the volume of data associated to your request
# synda show -s ./selection/...

Step 4: Generate file list associated to your request

by evaluating the cell below, the file list associated to your request at this time is generated. In case this fails please first try to debug your request using the cells above or by directly installing synda at your site. If problems persist please continue with the form submission below - we will try to resolve your request by direct interaction with our data managers ..



In [ ]:

    
form.file_list = dkrz_forms.utils.get_file_list(form.sel_file_list)

Step 5: Provide additional information with respect to your request

to be completed
info on:
- update frequency requested (new versions)
- when data can be deleted
- scientific/project context this data is needed for ...
- ...

Step 6: Submit your data replication request

Please provide the file names of the selection files you tested above and which you now want to submit to the DKRZ data managers.



In [ ]:

    
form_handler.save_form(sf,"..my comment..") # add a comment to remember this specific 


form_handler.email_form_info(sf)  # do not change
form_handler.form_submission(sf)  # do not change

Appendix: Example synda calls

play around with synda ..

Explore Metadata

example synda calls to search and explore metadata



In [ ]:

    
%%bash 

# synda dump tas GFDL-ESM2M -F line -f -C size,filenam
synda variable tas
# synda search cmip5 MOHC HadGEM2-A amip4
# synda search cmip5 mon atmos -l 1000xCO2 mon atmos Amon r1i1p1



In [ ]:

    
%%bash 

synda -h



In [ ]:

    
%%bash 

synda check selection -s selection/test.txt