In [1]:
from dmpipe.pipeline import Pipeline


/nfs/slac/kipac/fs1/u/dmcat/software/anaconda2/envs/dmcat-gitdev/lib/python2.7/site-packages/matplotlib/__init__.py:1350: UserWarning:  This call to matplotlib.use() has no effect
because the backend has already been chosen;
matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

  warnings.warn(_use_error_msg)
Using /nfs/slac/kipac/fs1/u/dmcat/software/anaconda2/envs/dmcat-gitdev/lib/python2.7/site-packages/dmsky-0.2.1-py2.7.egg/dmsky/data/targets for TargetLibrary
Using /gpfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/test_dmpipe/dmsky_data/targets for TargetLibrary

First we build a pipeline object to analyze the dSphs, and point it at the config file for the dSphs analysis.


In [2]:
pipe = Pipeline(linkname = 'dSphs')

In [3]:
configfile = 'config/master_dSphs.yaml'

We have to 'preconfigure' the pipeline, because we need to build up the list of targets so that we correctly set up the later stages of the pipeline.


In [4]:
pipe.preconfigure(configfile)


return code  0

We then tell the pipeline to update the arguments for all of the steps that comprise it.


In [5]:
pipe.update_args(dict(config=configfile))

Now we look around a bit, we can drill down into the links that make up the pipeline. In this case we have a pipeline that consists of a couple of preparation jobs, then 5 sub-pipelines ('data', 'sim_xxx', 'random'). The sub-pipeline analyze the real data, and some different simulation cases. The 'random' sub-pipeline analyzes random sky directions.

At this point the pipeline is fully configured, let's look around.


In [6]:
pipe.linknames


Out[6]:
['prepare-targets',
 'spec-table',
 'data',
 'sim_100GeV_bb_3e26',
 'sim_null',
 'sim_100GeV_bb_1e25',
 'random',
 'final-plots']

In [7]:
pipe['data']


Out[7]:
<dmpipe.pipeline.PipelineData at 0x7f8347d897d0>

In [8]:
pipe['data'].linknames


Out[8]:
['analyze-roi',
 'analyze-sed',
 'convert-castro',
 'stack-likelihood',
 'plot-castro-sg',
 'plot-dm-sg',
 'plot-limits-sg',
 'plot-stacked-dm-sg',
 'plot-stacked-limits-sg']

We can print the status for the various links. Using recurse=True will also drill down to the status of the links in the sub pipelines.


In [9]:
pipe.print_status()


                      Linkname :     Link Status :          Jobs Status
               prepare-targets :            Done :                 Done
                    spec-table :           Ready :              Pending
                          data :           Ready :              Pending
            sim_100GeV_bb_3e26 :           Ready :              Pending
                      sim_null :           Ready :              Pending
            sim_100GeV_bb_1e25 :           Ready :              Pending
                        random :           Ready :              Pending
                   final-plots :           Ready :    8/   0/   0/   0/   0/   8

In [10]:
pipe.print_status(recurse=True)


                      Linkname :     Link Status :          Jobs Status
               prepare-targets :            Done :                 Done
                    spec-table :           Ready :              Pending
                          data :           Ready :              Pending
----------                             data    -----------
                        Linkname :     Link Status :          Jobs Status
                     analyze-roi :           Ready :    2/   0/   0/   0/   0/   2
                     analyze-sed :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                  plot-castro-sg :           Ready :    2/   0/   0/   0/   0/   2
                      plot-dm-sg :           Ready :    8/   0/   0/   0/   0/   8
                  plot-limits-sg :           Ready :    8/   0/   0/   0/   0/   8
              plot-stacked-dm-sg :           Ready :    4/   0/   0/   0/   0/   4
          plot-stacked-limits-sg :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
            sim_100GeV_bb_3e26 :           Ready :              Pending
----------               sim_100GeV_bb_3e26    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
                      sim_null :           Ready :              Pending
----------                         sim_null    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
            sim_100GeV_bb_1e25 :           Ready :              Pending
----------               sim_100GeV_bb_1e25    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
                        random :           Ready :              Pending
----------                           random    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                  random-dir-gen :           Ready :    2/   0/   0/   0/   0/   2
                     analyze-sed :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
------------------------------------------------
                   final-plots :           Ready :    8/   0/   0/   0/   0/   8

We can access a particular Link in the pipeline.
We can ask: what the default options for the link are ('_options') what the current set of options are ('args')


In [11]:
pipe['data']['analyze-roi']


Out[11]:
<fermipy.jobs.target_analysis.AnalyzeROI_SG at 0x7f8348281210>

In [12]:
pipe['data']['analyze-roi']._options


Out[12]:
{'action': ('run', 'Action to perform', str),
 'check_status_once': (False,
  'Check status only once before proceeding',
  bool),
 'config': (None, 'Path to fermipy config file.', str),
 'dry_run': (False, 'Print commands, but do not execute them', bool),
 'job_check_sleep': (300,
  'Sleep time between checking on job status (s)',
  int),
 'make_plots': (False, 'Make plots', bool),
 'print_update': (False, 'Print summary of job status', bool),
 'roi_baseline': ('fit_baseline', 'Key for roi baseline file.', str),
 'targetlist': (None, 'Path to the target list.', str),
 'ttype': (None, 'Type of target being analyzed.', str)}

In [13]:
pipe['data']['analyze-roi'].args


Out[13]:
{'action': 'run',
 'check_status_once': False,
 'config': None,
 'dry_run': False,
 'job_check_sleep': 300,
 'make_plots': False,
 'print_update': False,
 'roi_baseline': 'fit_baseline',
 'targetlist': None,
 'ttype': 'dSphs'}

We can ask what jobs get run by this link. In this paritcular case the link runs 1 job that in turn dispatches several other jobs to the batch farm.


In [14]:
pipe['data']['analyze-roi'].jobs


Out[14]:
OrderedDict([('__top__@dSphs.data.analyze-roi',
              jobname   : dSphs.data.analyze-roi
                jobkey  : __top__
                appname : fermipy-analyze-roi-sg
                logfile : logs/dSphs.data.analyze-roi.log
                status  : Ready)])

Here we are talking to the link that represents any one of the disptached jobs. The command_template() function tells us how we would run this job from the UNIX command line.


In [15]:
pipe['data']['analyze-roi'].scatter_link


Out[15]:
<fermipy.jobs.target_analysis.AnalyzeROI at 0x7f83933d0f50>

In [16]:
pipe['data']['analyze-roi'].scatter_link.command_template()


Out[16]:
'fermipy-analyze-roi --roi_baseline {roi_baseline} --config {config}'

Here are ask what jobs will be dispatched. Note that there are two jobs with slightly different names.


In [17]:
pipe['data']['analyze-roi'].scatter_link.jobs


Out[17]:
OrderedDict([('draco@dSphs.data.analyze-roi',
              jobname   : dSphs.data.analyze-roi
                jobkey  : draco
                appname : fermipy-analyze-roi
                logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log
                status  : Ready),
             ('segue_1@dSphs.data.analyze-roi',
              jobname   : dSphs.data.analyze-roi
                jobkey  : segue_1
                appname : fermipy-analyze-roi
                logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/segue_1/analyze-roi_segue_1.log
                status  : Ready)])

Here are are asking for information about the first of those two jobs. In particular, the specific options used for this instance of the job. You can merge the job_config with the command_template to get the exact syntax for the instance of this command.


In [18]:
pipe['data']['analyze-roi'].scatter_link.jobs['draco@dSphs.data.analyze-roi']


Out[18]:
jobname   : dSphs.data.analyze-roi
  jobkey  : draco
  appname : fermipy-analyze-roi
  logfile : /nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log
  status  : Ready

In [19]:
pipe['data']['analyze-roi'].scatter_link.jobs['draco@dSphs.data.analyze-roi'].job_config


Out[19]:
{'config': './dSphs/draco/config.yaml',
 'logfile': '/nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/analyze-roi_draco.log',
 'make_plots': False,
 'roi_baseline': 'fit_baseline'}

Here we run a single link.
We use 'run_with_log()' so that the log file will be in the correct place and the pipeline will know the job has completed.


In [20]:
pipe['spec-table'].run_with_log()


return code  0

In [21]:
pipe.print_status()


                      Linkname :     Link Status :          Jobs Status
               prepare-targets :            Done :                 Done
                    spec-table :            Done :                 Done
                          data :           Ready :              Pending
            sim_100GeV_bb_3e26 :           Ready :              Pending
                      sim_null :           Ready :              Pending
            sim_100GeV_bb_1e25 :           Ready :              Pending
                        random :           Ready :              Pending
                   final-plots :           Ready :    8/   0/   0/   0/   0/   8

Here we run the 'data' sub-pipeline.


In [22]:
pipe['data'].run(resubmit_failed=True)


Running link  dSphs.data.analyze-roi
Submitting jobs (2): !
Checking status (300s): ...! Done
Running link  dSphs.data.analyze-sed
Submitting jobs (2): !
Checking status (300s): .! Done
Running link  dSphs.data.convert-castro
Submitting jobs (4): !
Checking status (120s): .! Done
Running link  dSphs.data.stack-likelihood
Submitting jobs (2): !
Checking status (60s): .! Done
Running link  dSphs.data.plot-castro-sg
Submitting jobs (2): !
Checking status (60s): .! Done
Running link  dSphs.data.plot-dm-sg
Submitting jobs (8): !
Checking status (60s): .! Done
Running link  dSphs.data.plot-limits-sg
Submitting jobs (8): !
Checking status (60s): .! Done
Running link  dSphs.data.plot-stacked-dm-sg
Submitting jobs (4): !
Checking status (60s): .! Done
Running link  dSphs.data.plot-stacked-limits-sg
Submitting jobs (4): !
Checking status (60s): .! Done
Chain status 5
Timestamp: 1527130057
Successfully completed

In [23]:
pipe.print_status(recurse=True)


                      Linkname :     Link Status :          Jobs Status
               prepare-targets :            Done :                 Done
                    spec-table :            Done :                 Done
                          data :            Done :                 Done
----------                             data    -----------
                        Linkname :     Link Status :          Jobs Status
                     analyze-roi :            Done :    0/   0/   0/   2/   0/   2
                     analyze-sed :            Done :    0/   0/   0/   2/   0/   2
                  convert-castro :            Done :    0/   0/   0/   4/   0/   4
                stack-likelihood :            Done :    0/   0/   0/   2/   0/   2
                  plot-castro-sg :            Done :    0/   0/   0/   2/   0/   2
                      plot-dm-sg :            Done :    0/   0/   0/   8/   0/   8
                  plot-limits-sg :            Done :    0/   0/   0/   8/   0/   8
              plot-stacked-dm-sg :            Done :    0/   0/   0/   4/   0/   4
          plot-stacked-limits-sg :            Done :    0/   0/   0/   4/   0/   4
------------------------------------------------
            sim_100GeV_bb_3e26 :           Ready :              Pending
----------               sim_100GeV_bb_3e26    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
                      sim_null :           Ready :              Pending
----------                         sim_null    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
            sim_100GeV_bb_1e25 :           Ready :              Pending
----------               sim_100GeV_bb_1e25    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                    simulate-roi :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
             plot-control-limits :           Ready :    4/   0/   0/   0/   0/   4
------------------------------------------------
                        random :           Ready :              Pending
----------                           random    -----------
                        Linkname :     Link Status :          Jobs Status
                   copy-base-roi :           Ready :    2/   0/   0/   0/   0/   2
                  random-dir-gen :           Ready :    2/   0/   0/   0/   0/   2
                     analyze-sed :           Ready :    2/   0/   0/   0/   0/   2
                  convert-castro :           Ready :    4/   0/   0/   0/   0/   4
                stack-likelihood :           Ready :    2/   0/   0/   0/   0/   2
                     collect-sed :           Ready :    2/   0/   0/   0/   0/   2
                  collect-limits :           Ready :    4/   0/   0/   0/   0/   4
          collect-stacked-limits :           Ready :    2/   0/   0/   0/   0/   2
                 plot-stacked-dm :           Ready :   80/   0/   0/   0/   0/  80
             plot-stacked-limits :           Ready :   80/   0/   0/   0/   0/  80
------------------------------------------------
                   final-plots :           Ready :    8/   0/   0/   0/   0/   8

In [24]:
pipe['data']['convert-castro'].scatter_link.jobs['draco:ack2016_point:lgauss@dSphs.data.convert-castro'].job_config


Out[24]:
{'clobber': False,
 'j_value_file': './dSphs/draco/j_val_ack2016.yaml',
 'jprior': 'lgauss',
 'limitfile': './dSphs/draco/dmlimits_ack2016_point_lgauss.fits',
 'logfile': '/nfs/slac/kipac/fs1/u/dmcat/workdir/sandbox/dmpipe_example/dSphs/draco/dmlike_ack2016_point_lgauss.log',
 'nsims': -1,
 'outfile': './dSphs/draco/dmlike_ack2016_point_lgauss.fits',
 'sed_file': './dSphs/draco/sed_ack2016_point.fits',
 'seed': -1,
 'specfile': './dm_spectra_dSphs.fits'}

In [25]:
pipe.run(resubmit_failed=True)


Skipping done link dSphs.prepare-targets
Skipping done link dSphs.spec-table
Running link  dSphs.data.data
Skipping done link dSphs.data.analyze-roi
Skipping done link dSphs.data.analyze-sed
Skipping done link dSphs.data.convert-castro
Skipping done link dSphs.data.stack-likelihood
Skipping done link dSphs.data.plot-castro-sg
Skipping done link dSphs.data.plot-dm-sg
Skipping done link dSphs.data.plot-limits-sg
Skipping done link dSphs.data.plot-stacked-dm-sg
Skipping done link dSphs.data.plot-stacked-limits-sg
Chain status 5
Running link  dSphs.sim_100GeV_bb_3e26.sim_100GeV_bb_3e26
Running link  dSphs.sim_100GeV_bb_3e26.copy-base-roi
Submitting jobs (2): !
Checking status (60s): .! Done
Running link  dSphs.sim_100GeV_bb_3e26.simulate-roi
Submitting jobs (2): !
Checking status (300s): ......! Done
Running link  dSphs.sim_100GeV_bb_3e26.convert-castro
Submitting jobs (4): !
Checking status (120s): ...! Done
Running link  dSphs.sim_100GeV_bb_3e26.stack-likelihood
Submitting jobs (2): !
Checking status (60s): ..! Done
Running link  dSphs.sim_100GeV_bb_3e26.collect-sed
Submitting jobs (2): !
Checking status (60s): .! Done
Running link  dSphs.sim_100GeV_bb_3e26.collect-limits
Submitting jobs (4): !
Checking status (60s): ...! Partially Failed
Stoping chain execution at failed link dSphs.sim_100GeV_bb_3e26.collect-limits
Chain status 7
Stoping chain execution at failed link dSphs.sim_100GeV_bb_3e26.sim_100GeV_bb_3e26
Chain status 7
Timestamp: 1527132646
Exited with exit code -1

In [ ]: