Introduction to the Merged Target List

Author: Adam D. Myers, University of Wyoming

This Notebook describes how the logic in mtl.py (the Merged Target List) uses priorities and numbers of observations set by the targeting bitmasks to determine the observational "state" of a target.

If you identify any errors or have requests for additional functionality please create a new issue at https://github.com/desihub/desitarget/issues or send a note to desi-data@desi.lbl.gov.

Last updated May 2020 using DESI software release:

Getting started

Using NERSC

The easiest way to get started is to use the jupyter server at NERSC so that you don't need to install any code or download any data locally.

If you need a NERSC account, see https://desi.lbl.gov/trac/wiki/Computing/AccessNersc

Then do the one-time jupyter configuration described at https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC

From a NERSC command line, checkout a copy of the tutorial code, e.g. from cori.nersc.gov

mkdir -p $HOME/desi/
cd $HOME/desi/
git clone https://github.com/desihub/tutorials

And then go to https://jupyter.nersc.gov, login, navigate to where you checked out this package (e.g. $HOME/desi/tutorials), and double-click on Intro_to_mtl.ipynb.

This tutorial has been tested using the:

kernel installed at NERSC. To get an equivalent environment from a cori command line use the command printed out below:

Import required modules

Note that we will focus on the Main Survey targeting bit mask for this tutorial. If you instead wanted to study the behavior of MTL for the SV bit-mask, you would use:

from desitarget.sv1.sv1_targetmask import desi_mask

If you are running locally and any of these fail, you should go back through the installation instructions and/or email desi-data@desi.lbl.gov if you get stuck. If you are running from jupyter.nersc.gov and have problems, double check that your kernel is as printed below:

Preliminaries

This tutorial will focus on aspects of the targeting bitmasks and coding logic that control the observational state of a target. To understand the more basic aspects of the targeting bitmasks (how we know what is a QSO, or ELG, target, etc.) you might want to try the bits and bitmasks tutorial.

How a target is observed (how it is handled by the desitarget fiberassign code) depends on three main targeting concepts:

• The observational layer (conditions) in which a target can be observed. For instance, is a target suitable for bright-time or dark-time obervations?
• The priority of the target. Is this target a highest-priority target, or should we place a fiber on another target first instead?
• The number of observations for the target. Should we place a fiber on this target 1 time? Or maybe 4 times?

Initially, all of these quantities are set by static information in the desitarget bitmask yaml file but they can subsequently be altered by the desitarget mtl code depending on each target's observational state.

Understanding what information is intially set for each targeting bit

When a target is initally flagged for DESI observations, the information in the desitarget bitmask yaml file is used to set an initial observational layer, priority and number of observations. Consider the following example for a quasar target:

Or for an ELG or LRG target:

These quantities define the "state-of-play" for a given target before any DESI observations have occurred. So, for example, if an ELG is being observed for the first time, we will request 1 observation of that target at a priority of 3000, if a QSO is being observed for the first time, we will request 4 observations of such a target at a priority of 3400. The actual relative values of the priorites are unimportant, but the target with the highest priority will always be assigned a fiber first.

The values in the priorities dictionary merit some further explanation. As we will see later in this notebook, the priority of a target can change depending on its observational state. The keys in the priorities dictionary define what priority to set for a target that has transitioned to that observational state.

So, for example, an LRG will start with a priority of 3200 (UNOBS) but will transition to a priority of 2 (DONE) as soon as we have one observation of it. For target classes that request more than one observation, such as Lyman-alpha QSOs, the behavior can be more complex. If we have observed the QSO and request more observations, and the redshift is flagged as problematic, the QSO target will be assigned a priority of 3400 (MORE_ZWARN). If we have observed the QSO and request more observations, and the redshift is flagged as good, the QSO target will be assigned a priority of 3500 (MORE_ZGOOD). The full suite of allowed observational states for a DESI target can be retrieved using the obsmask bitmask (which we imported earlier):

Similarly, the full suite of allowed observational layers (dark-time, bright-time, etc.) can be retrieved using the obsconditions bitmask:

What happens if a target is both an ELG and a QSO?

No target is an island, and it is certainly possible for a DESI target to satisfy two sets of target selection criteria and be assigned a bit value consistent with two targets. Again, you might want to try the bits and bitmasks tutorial. for more insight into bit values. For example, though, consider the first 8 possible bit values and what they signify for the various DESI target classes:

Bit value (2 + 4 =) 6, for instance, which corresponds to 2¹ ("ELG") + 2² ("QSO"), denotes a target that satisfies the selection criteria for two different target classes. How do we set priorities and numbers of observations in such a case?

Clearly, these targets need to have their priorities "merged" in some sense. If you've been paying attention, you've probably worked out that this is part of what the desitarget mtl (Merged Target List; henceforth MTL) code achieves.

Let's set up a specific example for a target that is both an ELG and QSO, and see how MTL processes such a target.

So, now we have a simple set of targets defined. We'll also need to add some standard columns, as these are expected by the MTL code:

Let's see what happened to the priorities and observing conditions (layers) for these targets after they were merged by calling the MTL code:

Some important things to note:

• The priority for the merged "ELG/QSO" target was set to that of the highest priority target.
• The observing conditions for the merged "ELG/QSO" target were combined across all targets.
• As the logic in MTL is different depending on the observing layer, the MTL code expects to be passed an observing layer to understand what "flavor" of survey (dark-time, etc.) it is processing.
  • Currently, this functionality means that users will need to pass either obscon="DARK|GRAY" (for the dark-time survey) or obscon="BRIGHT" (for the bright-time survey).
  • It's entirely possible, though, that "special" layers with unique MTL logic could be created in the future (triggered by, e.g., obscon="SOME_SPECIAL_LAYER").

Note that this example was purely to show you why a mechanism for merging targets is critical. In reality, desitarget sets initial priorities, observing conditions, and numbers of observations in advance of running MTL. This makes the initial values of these parameters more traceable (as they then appear as column names in the desitarget initial targeting files). Critically, this step has to be done to correctly initialize the numbers of observations (NUMOBS_INIT) for DESI targets. So a more complete example is:

Note, for instance, that the merged "ELG/QSO" target requires 4 observations (as it is potentially a Lyman-alpha QSO target).

In "official" DESI targeting files, e.g. as stored in the following NERSC directories:

/global/cfs/projectdirs/desi/target/catalogs/dr8
/global/cfs/projectdirs/desi/target/catalogs/dr9

PRIORITY_INIT and NUMOBS_INIT have already been set in this manner.

With reasonable initial values of PRIORITY_INIT and NUMOBS_INIT, MTL will pass through the number of additional observations required for each target (NUMOBS_MORE). For example:

Updating the status of a target

The MTL code also contains logic to update the priorities of, and number of observations for, a target based on each target's current observational state. Most of this logic is contained in the desitarget.targets module.

As the DESI survey progresses, classifications and redshifts of each target will be included in a redshift catalog (henceforth a zcat) passed back to MTL from the DESI spectroscopic pipeline. Passing this zcat as an input to MTL, changes the observational state of each target, updating the number of required additional observations and transitioning between the priorites described earlier in this tutorial ("UNOBS", "DONE", etc.). Let's look at an example. First, let's construct a set of targets in a manner similar to what we did in the previous section of this tutorial:

Now, let's also construct a zcat:

So, here's the initial list of target properties (a static set of assignations):

and here's the spectroscopic information we gleaned from observing these targets once in DESI:

Let's see what MTL makes of all of this:

To summarize the output:

• Any target (ELGs, LRGs) for which only one observation was requested has had its priority set to the equivalent of "DONE".
• Any QSO target that was categorically confirmed to be a ("tracer") QSO at z < 2.15 without any warnings has had NUMOBS_MORE set to 0 and its priority set to the equivalent of "DONE".
• Any QSO target that was confirmed to be a ("Lyman-alpha") QSO at z > 2.15 without any warnings has had its priority set to the equivalent of "MORE_ZGOOD" and NUMOBS_MORE is set to 3.
• Any QSO target for which the spectrum flagged a redshift warning has had NUMOBS_MORE set to 3 but has retained its initial priority of 3400.

How the MTL logic affects every (dark-time) target class

By extension of the last example, we can test how MTL affects every target class individually, remembering the general precepts that for merged targets (e.g. a target that is both an ELG and a QSO) the highest PRIORITY and NUMOBS for the individual classes will characterize the behavior, and all observing conditions (layers) will be merged across classes. Here goes: