Gather unWISE photometry for the HST sample

The goal of this notebook is to document how we retrieve DR14 SDSS ugriz and unWISE W1-W4 (forced) photometry for the HST HizEA sample of galaxies.

The input catalog is

  • $HIZEA_DIR/etc/hst_sample.dat

and the output (row-matched) catalog containing just the SDSS/unWISE photometry is

  • $HIZEA_DIR/etc/XXX

In [34]:
import os
import numpy as np
import matplotlib.pyplot as plt

In [35]:
import fitsio
from astropy.io import ascii
from astropy.table import Table

In [36]:
import astropy.units as u
from astropy.coordinates import SkyCoord

In [37]:
%matplotlib inline

In [20]:
def read_hizea():
    cat = ascii.read( os.path.join( os.getenv('HIZEA_DIR'), 'etc', 'hst_sample.dat' ) )
    return cat

In [21]:
hizea = read_hizea()
hizea


Out[21]:
<Table length=29>
galaxyradeczr_evoutwise_sfr
str10float64float64float64float64int64int64
J1506+5402226.6512396254.039099070.6080.13-1211621
J0905+5759136.348339457.986800930.7120.1-2470344
J1341-0321205.40333219-3.357018660.6580.16-875490
J0944+0930146.074369369.505378160.5140.15-1778161
J2140+1209325.0020524812.154058480.7520.23-606378
J0826+4305126.6600715343.091507380.6030.31-1228246
J1613+2834243.3855276128.57077320.4490.68-2416461
J1219+0336184.982412333.604416870.4510.76-1830253
J1107+0417166.76196744.284101870.4670.56-1828118
J0901+0314135.389276073.236797320.4590.51-120692
.....................
J2256+1504344.099005715.073546660.7270.76-8008
J1506+6131226.5153918961.530034980.4370.8308
J0908+1039137.1781804110.650218680.5021.2408
J0827+2954126.8911455329.91423670.6811.3608
J1634+4619248.6936658846.329678570.5761.79011
J1232+0723188.065946917.389092120.42.2-49115
J1104+5946166.1560959259.777677360.5730.2-13540
J1229+3545187.4576318735.754754340.6140.95-6690
J1248+0601192.029828046.019956820.6320.78-4770
J1713+2817258.2516060228.285630740.5770.74-8680

In [27]:
def read_unwise():
    unwisefile = os.path.join(os.getenv('IM_DATA_DIR'), 'unwise', 'specmatch-dr10.fits')
    ff = fitsio.FITS(unwisefile)
    columns = list( ('wise_ra', 'wise_dec') )
    columns = columns + ['wise_{}_nanomaggies'.format(band) for band in ('w1', 'w2', 'w3', 'w4')]
    columns = columns + ['wise_{}_nanomaggies_ivar'.format(band) for band in ('w1', 'w2', 'w3', 'w4')]
    print('Reading {}'.format(unwisefile))
    ww = ff[1].where("wise_ra > 0 && wise_dec > 0")
    unwise = Table(ff[1].read(columns=columns, ext=1, rows=ww))
    return unwise

In [28]:
unwise = read_unwise()
unwise


Reading /Users/ioannis/research/data/unwise/specmatch-dr10.fits
Out[28]:
<Table length=2683302>
wise_rawise_decwise_w1_nanomaggieswise_w1_nanomaggies_ivarwise_w2_nanomaggieswise_w2_nanomaggies_ivarwise_w3_nanomaggieswise_w3_nanomaggies_ivarwise_w4_nanomaggieswise_w4_nanomaggies_ivar
float64float64float32float32float32float32float32float32float32float32
146.9438812220.03917583515211363.250.005007471358.720.000374917-839.8719.20983e-08174809.01.39546e-10
147.3295019050.028900754039211432.10.001391813918.10.00011565496630.03.14952e-088.2514e+066.14016e-11
146.7638354890.0414016530208469.0980.004147871412.710.00030616224892.56.6069e-0860771.79.65525e-11
146.7607690380.01595391882644319.50.002014725020.110.00015421108112.03.84801e-08480762.06.80027e-11
146.8091359970.02636290837527506.410.001551288952.060.00012238714974.53.01488e-08641314.05.72089e-11
146.7290088570.02695727354493413.850.003092813694.310.0002396291511.05.21995e-0858216.68.79149e-11
146.6307001680.07097453849692902.070.002060023358.640.0001616013242.643.9688e-08152124.07.36789e-11
146.6080705140.04767824464851341.390.002509741631.970.00019158738923.74.87348e-08144250.08.31926e-11
146.3698720980.08218923997412359.740.00227752520.180.00018082363687.53.84821e-08177877.06.80085e-11
146.3470768290.07540947334222731.890.002542312806.180.00019514341691.74.22733e-08232807.07.19846e-11
..............................
259.74290341641.7404369548599.1190.0109069721.0380.0008585212133.912.1519e-07-15407.32.85316e-10
259.91840576341.678259023975.12930.0105713149.0170.0008361092909.891.78656e-0722914.31.53483e-10
259.9129649941.7032820724128.3060.00983937-10.05280.0007675298542.91.75821e-07-310130.01.59183e-10
259.71618974341.80462751541722.080.007696532103.80.000607387-3320.921.61518e-07-24623.62.42267e-10
259.64528799541.6499484663709.7810.00742951792.360.000596938-5033.451.5972e-07254539.02.43556e-10
259.63548054941.61974234522506.340.004527113123.170.000398389-1950.261.1186e-07-175478.01.95864e-10
259.65943955441.75823096451055.20.008013241137.680.000657905-940.6021.72005e-0738971.62.53323e-10
259.78872848241.76901514671730.930.006014912022.310.0005016045348.921.41012e-07-179993.02.20564e-10
259.76985476441.63972056481067.730.008210041145.50.000662173-4834.611.66114e-07-94671.62.41504e-10
259.68628317541.8085103216821.5310.00881233835.8930.000675509-829.0771.72092e-07-62637.62.38301e-10

In [57]:
unwisecoord = SkyCoord(ra=unwise['wise_ra']*u.deg, dec=unwise['wise_dec']*u.deg)
hizeacoord = SkyCoord(ra=hizea['ra']*u.deg, dec=hizea['dec']*u.deg)
idx, sep2d, d3d = hizeacoord.match_to_catalog_sky(unwisecoord, nthneighbor=1)

In [67]:
_ = plt.hist(3600*(unwise['wise_ra'][idx] - hizea['ra']).data)



In [54]:
def qa_sep(sep2d):
    fig, ax = plt.subplots()
    _, _, _ = ax.hist(sep2d.arcsec, bins=100, range=(0, 0.5), alpha=0.5)
    ax.set_xlabel('Separation (arcsec)')
    ax.set_ylabel('Number of Galaxies')

In [55]:
qa_sep(sep2d)



In [44]:
rad = 0.5 * u.arcsec

In [45]:
indx_unwise, indx_hizea, d2d, _ = hizeacoord.search_around_sky(unwisecoord, rad)
nmatch = len(indx_unwise)
print('Found {}/{} objects with matching unWISE photometry within {}'.format(nmatch, len(unwise), rad))


Found 29/2683302 objects with matching unWISE photometry within 0.5 arcsec

In [48]:
indx_hizea


Out[48]:
array([ 1,  1,  1,  7,  9,  8,  8, 20, 23, 14,  4, 19,  5,  5,  0, 27, 25,
       28, 13, 15, 12, 22,  3, 18,  6, 24, 17, 21, 26])

In [49]:
d2d


Out[49]:
[$0^\circ00{}^\prime00.0488{}^{\prime\prime}$ $0^\circ00{}^\prime00.0488{}^{\prime\prime}$ $0^\circ00{}^\prime00.0488{}^{\prime\prime}$ $0^\circ00{}^\prime00.0252{}^{\prime\prime}$ $0^\circ00{}^\prime00.0598{}^{\prime\prime}$ $0^\circ00{}^\prime00.0132{}^{\prime\prime}$ $0^\circ00{}^\prime00.0132{}^{\prime\prime}$ $0^\circ00{}^\prime00.0229{}^{\prime\prime}$ $0^\circ00{}^\prime00.0176{}^{\prime\prime}$ $0^\circ00{}^\prime00.0085{}^{\prime\prime}$ $0^\circ00{}^\prime00.034{}^{\prime\prime}$ $0^\circ00{}^\prime00.0602{}^{\prime\prime}$ $0^\circ00{}^\prime00.0438{}^{\prime\prime}$ $0^\circ00{}^\prime00.0438{}^{\prime\prime}$ $0^\circ00{}^\prime00.0182{}^{\prime\prime}$ $0^\circ00{}^\prime00.0796{}^{\prime\prime}$ $0^\circ00{}^\prime00.0312{}^{\prime\prime}$ $0^\circ00{}^\prime00.0018{}^{\prime\prime}$ $0^\circ00{}^\prime00.0292{}^{\prime\prime}$ $0^\circ00{}^\prime00.0062{}^{\prime\prime}$ $0^\circ00{}^\prime00.0286{}^{\prime\prime}$ $0^\circ00{}^\prime00.0236{}^{\prime\prime}$ $0^\circ00{}^\prime00.0267{}^{\prime\prime}$ $0^\circ00{}^\prime00.0876{}^{\prime\prime}$ $0^\circ00{}^\prime00.0107{}^{\prime\prime}$ $0^\circ00{}^\prime00.0364{}^{\prime\prime}$ $0^\circ00{}^\prime00.1205{}^{\prime\prime}$ $0^\circ00{}^\prime00.0256{}^{\prime\prime}$ $0^\circ00{}^\prime00.0171{}^{\prime\prime}$]

In [46]:
unwise[indx_unwise]


Out[46]:
<Table length=29>
wise_rawise_decwise_w1_nanomaggieswise_w1_nanomaggies_ivarwise_w2_nanomaggieswise_w2_nanomaggies_ivarwise_w3_nanomaggieswise_w3_nanomaggies_ivarwise_w4_nanomaggieswise_w4_nanomaggies_ivar
float64float64float32float32float32float32float32float32float32float32
136.34832204657.9867909564473.6540.0071384567.1860.00056941619708.21.10561e-07287461.01.53568e-10
136.34832204657.9867909564473.6540.0071384567.1860.00056941619708.21.10561e-07287461.01.53568e-10
136.34832204657.9867909564473.6540.0071384567.1860.00056941619708.21.10561e-07287461.01.53568e-10
184.9824053823.60441774002979.8070.004802861336.840.00027052255407.05.80838e-08752576.01.12701e-10
135.3892596133.23679973794799.9280.00387882989.3320.00029145721333.86.14208e-08326297.08.95338e-11
166.7619685074.28409836516582.7810.004339661069.950.00031596449007.17.5229e-08499158.01.13078e-10
166.7619685074.28409836516582.7810.004339661069.950.00031596449007.17.5229e-08499158.01.13078e-10
226.51538161161.53003093519.8750.0136851614.1660.0010746513966.12.91995e-0740246.03.95725e-10
248.6936717246.32968135231526.030.01192782301.250.00091674919253.23.93181e-07-90411.82.63432e-10
248.92380425847.1567862719854.9170.0119446933.2120.0009349357655.264.11701e-07106613.03.09594e-10
..............................
209.8374367851.6274841105660.1530.00894641120.10.00068944117256.11.78757e-07133373.02.54092e-10
239.54683130739.9557877682867.310.007858631470.980.00064675451978.72.8178e-07359837.03.96647e-10
126.89115159929.9142405927507.9290.00424538588.7080.0003472247727.557.11581e-08148391.01.07161e-10
146.0743702019.50538552106628.2920.00369038842.8260.00027515531190.75.98348e-08402505.08.90742e-11
159.77903192845.6316812323963.2790.005984221139.290.0004410959515.481.01054e-07-20407.91.43664e-10
243.38552435428.57077234741145.230.007344621878.010.00057678982461.71.50547e-071.20294e+061.58354e-10
188.0659393367.38908536904830.7590.003369471240.090.00020167319456.34.32566e-08-616781.05.13074e-11
222.62024780746.3604735686834.1240.00870505899.3320.00068802611700.72.01726e-07125266.02.69918e-10
137.17817674810.65021254161105.370.00451771425.450.00033188514264.08.01954e-0830398.61.23033e-10
187.45762791335.7547578247846.5350.0048696889.9680.0003560565910.258.31471e-0815085.71.15093e-10

[1] Read the sample and write out a temporary file we can upload to CasJobs.


In [ ]:


In [2]:
def write_casjobs_file():
    outfile = os.path.join(os.sep, 'Users', 'ioannis', 'Desktop', 'hizea_sample.tmp')
    dat = ascii.read( os.path.join( os.getenv('HIZEA_DIR'), 'etc', 'hst_sample.dat' ) )
    out = Table()
    out['hizea_id'] = np.arange(len(dat))
    out['hizea_ra'] = dat['ra']
    out['hizea_dec'] = dat['dec']
    print('Writing {}'.format(outfile))
    out.write(outfile, format='ascii.commented_header', overwrite=True)
    return out

In [3]:
out = write_casjobs_file()
out


Writing /Users/ioannis/Desktop/hizea_sample.tmp
Out[3]:
<Table length=29>
hizea_idhizea_rahizea_dec
int64float64float64
0226.6512396254.03909907
1136.348339457.98680093
2205.40333219-3.35701866
3146.074369369.50537816
4325.0020524812.15405848
5126.6600715343.09150738
6243.3855276128.5707732
7184.982412333.60441687
8166.76196744.28410187
9135.389276073.23679732
.........
19344.099005715.07354666
20226.5153918961.53003498
21137.1781804110.65021868
22126.8911455329.9142367
23248.6936658846.32967857
24188.065946917.38909212
25166.1560959259.77767736
26187.4576318735.75475434
27192.029828046.01995682
28258.2516060228.28563074

[2] Import the sample into CasJobs.

Unfortunately when you import a simple ASCII file, CasJobs does not keep the RA,Dec coordinates in double precision. So first create a table in the Query tab (using the Quick submission option) thusly:

CREATE TABLE hizea ( hizea_id int, hizea_ra float, hizea_dec float );

and then in the Import tab be sure to import the data directly into this table.

[3] Search for ObjID using RA,Dec positions.

Next, gather the SDSS ObjID using the RA,Dec positions and a 1 arcsec search radius. In the Query tab be sure to set the Context to DR14 and then execute the following query.

SELECT
  h.hizea_id,
  h.hizea_ra,
  h.hizea_dec,
  p.objid,
  p.ra,
  p.dec,
  p.run,
  p.rerun,
  p.camcol
INTO mydb.hizea_match FROM mydb.hizea AS h
  OUTER APPLY dbo.fGetNearestObjEq( h.hizea_ra, h.hizea_dec, 0.01666) AS n 
  LEFT JOIN dr14.PhotoObj AS p ON n.objid=p.objid
ORDER BY h.hizea_id

[4] Finally gather the SDSS + unWISE photometry.

Finally, we use the output of the previous step to retrieve the full set of SDSS (ugriz) and unWISE (W1W2) forced photometry by executing the following query (again, using the DR14 Context). Note that we call the output table (not creatively) hst_sample_sdssWISEphot.fits.

SELECT
  h.hizea_id,
  p.objid,
  p.ra,
  p.dec,
  p.run,
  p.rerun,
  p.camcol,
  p.field,
  p.type,
  p.petroflux_u,
  p.petroflux_g,
  p.petroflux_r,
  p.petroflux_i,
  p.petroflux_z,
  p.petrofluxivar_u,
  p.petrofluxivar_g,
  p.petrofluxivar_r,
  p.petrofluxivar_i,
  p.petrofluxivar_z,
  p.modelflux_u,
  p.modelflux_g,
  p.modelflux_r,
  p.modelflux_i,
  p.modelflux_z,
  p.modelfluxivar_u,
  p.modelfluxivar_g,
  p.modelfluxivar_r,
  p.modelfluxivar_i,
  p.modelfluxivar_z,
  p.cmodelflux_u,
  p.cmodelflux_g,
  p.cmodelflux_r,
  p.cmodelflux_i,
  p.cmodelflux_z,
  p.cmodelfluxivar_u,
  p.cmodelfluxivar_g,
  p.cmodelfluxivar_r,
  p.cmodelfluxivar_i,
  p.cmodelfluxivar_z,
  p.extinction_u,
  p.extinction_g,
  p.extinction_r,
  p.extinction_i,
  p.extinction_z,
  p.petroR50_u,
  p.petroR50_g,
  p.petroR50_r,
  p.petroR50_i,
  p.petroR50_z,
  f.w1_nanomaggies,
  f.w1_nanomaggies_ivar,
  f.w2_nanomaggies,
  f.w2_nanomaggies_ivar
INTO mydb.hst_sample_sdssWISEphot from mydb.hizea_match as h
  JOIN dr14.PhotoObj AS p ON h.objid=p.objid
  JOIN dr14.wiseForcedTarget AS f ON h.objid=f.objid
ORDER BY h.hizea_id

In [ ]: