Applying depth cuts to CCD files for DR8

I (Adam Myers) have not tested these instructions, I'm merely relating something Dustin Lang wrote in an email so that it's recorded for posterity.

Making depth cuts for DR8

The script for making depth cuts is legacyanalysis/depth-cut.py, and the legacyanalysis/depthcut.sh script can be used to set environment variables and give it args. That script will need to be edited with appropriate input and output paths. This gets run on each brick, producing a ccds-BRICK.fits file for each brick. Running it on each brick can be done using qdo. For more information on using qdo to process bricks see, e.g., Martin Landriau's cookbook or Adam Myers' tutorial.

After that finishes for each brick (this takes some time....), run the legacyanalysis/depth-cut-dr8.py script (again, modifying paths appropriately). The other "secret" step is converting the survey-ccds.fits files into the KD-tree-ified files using the legacypipe/create_kdtrees.py script.

Further details, taken from the header of the depth cut script:

Start with CCDs tables / zeropoints files.
Create survey-ccd-*.kd.fits files via
  python legacypipe/create-kdtrees.py

Create $CSCRATCH/dr8new containing:
calib
images
survey-bricks.fits.gz
survey-ccds-decam-g.kd.fits
survey-ccds-decam-r.kd.fits
survey-ccds-decam-z.kd.fits

Create "depthcut" qdo queue:
LEGACY_SURVEY_DIR=$CSCRATCH/dr8new python -u legacypipe/queue-calibs.py --region dr8-decam > bricks-decam.txt

(hand-edit off the first few chatter lines)

qdo load depthcut bricks-decam.txt

Run "depth-cut.py" on each brick:
QDO_BATCH_PROFILE=cori-shifter qdo launch -v depthcut 32 --cores_per_worker 1 --walltime=30:00 --batchqueue=debug --keep_env --batchopts "--image=docker:dstndstn/legacypipe:intel" --script "/src/legacypipe/py/legacyanalysis/depthcut.sh"

Then run the depth cut script itself.

Making depth cuts in the context of a full processing run

To summarize all of the steps needed to create calibration files and to use those files to process bricks:

  • legacyzpts/legacy_zeropoints.py takes filename to *-legacypipe.fits files.
  • legacyzpts/legacy_zeropoints_merge.py takes a set of *-legacypipe.fits files to a survey-ccds.fits file.
  • legacypipe/create_kdtrees.py takes survey-ccds.fits to survey-ccds-init.kd.fits.
  • build an initial LEGACY_SURVEY_DIR containing that survey-ccds-init.kd.fits.
  • legacypipe/queue-calibs.py creates a list of bricks (eg to feed qdo).
  • legacyanalysis/depthcut.sh takes bricks to ccds-BRICK.fits files.
  • legacyanalysis/depth-cut-dr8.py takes all the ccds-BRICK.fits files to a survey-ccds-depthcut.fits file.
  • legacypipe/create_kdtrees.py takes that survey-ccds-depthcut.fits file to survey-ccds-depthcut.kd.fits.
  • create a final LEGACY_SURVEY_DIR containing that new survey-ccds-depthcut.kd.fits.
  • legacypipe/runbrick.py on all bricks (see this notebook).