Flattening echelle spectra

Often-times, the model spectrum is just not good enough to get an adequate fit using the normal Fitters.RVFitter. When that is the case, the automated flattening just doesn't work. Let's give this a shot without using models. The thing we want to minimize is the offset between the flux at one order and the flux in the adjacent order.

$$L = \sum_{i=1}^{N_{orders}-1} \left(\frac{O_i(\lambda)}{M(i, \lambda| \Theta)} - \frac{O_{i+1}(\lambda)}{M(i+1, \lambda | \theta)}\right)^2 $$

Where $M(i, \lambda | \theta)$ is perhaps a 2d polynomial function of some order where $i$ is for the order number, $\lambda$ is the wavelength (or maybe pixel number?), and the $\theta$ are all the polynomial coefficients.

TRY:

Try using Robust Linear Models from statsmodels. That might help it to ignore the big Balmer line. I will need to put the Chebyshev factors into a feature vector to use this...

Those might help a bit. How about a hybrid approach where I divide by a stellar model (or empirical spectrum of similar type as Adam suggests), and then do a full robust 2D fit.

Robust linear models works well enough for me!

I was able to sufficiently flatten a few of my spectra to get RV fits. I need to calibrate the RV though, so pretty much need to flatten every spectrum and fit the RV. This will take a while...


In [1]:
import SpecFlattener
import glob
import StarData
from astropy.io import fits
import SpectralTypeRelations
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)


Module 'anfft' (FFTW Python bindings) could not be imported.
To install it, try running 'easy_install anfft' from the terminal.
Falling back on the slower 'fftpack' module for ND Fourier transforms.
:0: FutureWarning: IPython widgets are experimental and may change in the future.

In [2]:
#hdf5_lib = '/media/ExtraSpace/Kurucz_FullGrid/CHIRON_grid_full.hdf5'
hdf5_lib = '/Volumes/DATADRIVE/Kurucz_Grid/IGRINS_grid_full.hdf5'
star_list = [f for f in glob.glob('../201*/*corrected.fits') if 'flattened' not in f and 'oph' not in f.lower() and 'HER' not in f]
print(len(star_list))
#star_list.index('../20131019/HIP_22913.fits')
for s in star_list:
    print s


278
../20150410/HIP_70327_telluric_corrected.fits
../20140709/HIP_97376_telluric_corrected.fits
../20140709/HIP_85537_telluric_corrected.fits
../20140709/HIP_94620_telluric_corrected.fits
../20140709/HIP_93713_telluric_corrected.fits
../20140709/HIP_93580_telluric_corrected.fits
../20140709/HIP_95560_telluric_corrected.fits
../20140709/HIP_89935_telluric_corrected.fits
../20140709/HIP_93580_1_telluric_corrected.fits
../20140709/HIP_85385_telluric_corrected.fits
../20140709/HIP_90762_telluric_corrected.fits
../20150803/HIP_5518_telluric_corrected.fits
../20150803/HIP_2912_telluric_corrected.fits
../20141015/HIP_26126_telluric_corrected.fits
../20141015/HIP_22833_telluric_corrected.fits
../20141015/HIP_30666_telluric_corrected.fits
../20141015/HIP_24902_telluric_corrected.fits
../20141015/HIP_109056_5_telluric_corrected.fits
../20141015/HIP_14764_telluric_corrected.fits
../20141015/HIP_109056_1_telluric_corrected.fits
../20141015/HIP_93225_telluric_corrected.fits
../20141015/HIP_16611_telluric_corrected.fits
../20141015/HIP_29735_telluric_corrected.fits
../20141015/HIP_109056_4_telluric_corrected.fits
../20141015/HIP_85290_telluric_corrected.fits
../20141015/HIP_31278_telluric_corrected.fits
../20141015/HIP_100907_telluric_corrected.fits
../20141015/HIP_26093_telluric_corrected.fits
../20141015/HIP_20789_telluric_corrected.fits
../20141015/ADS_3962_AB_telluric_corrected.fits
../20141015/HIP_109056_telluric_corrected.fits
../20141015/HIP_23916_telluric_corrected.fits
../20141015/HIP_16322_telluric_corrected.fits
../20141015/HIP_109056_2_telluric_corrected.fits
../20141015/HIP_116611_telluric_corrected.fits
../20141015/HIP_116631_telluric_corrected.fits
../20141015/HIP_25790_telluric_corrected.fits
../20141015/HIP_12706_telluric_corrected.fits
../20141015/HIP_27713_telluric_corrected.fits
../20141015/HIP_15110_telluric_corrected.fits
../20141015/HIP_23362_telluric_corrected.fits
../20141015/HIP_105891_telluric_corrected.fits
../20141015/HIP_29151_telluric_corrected.fits
../20141015/HIP_109056_3_telluric_corrected.fits
../20141015/HIP_103298_telluric_corrected.fits
../20141015/HIP_25143_telluric_corrected.fits
../20150922/HIP_117089_telluric_corrected.fits
../20150922/HIP_111056_telluric_corrected.fits
../20150922/HIP_109831_telluric_corrected.fits
../20150922/HIP_116631_telluric_corrected.fits
../20150922/HIP_3478_telluric_corrected.fits
../20150411/HIP_72552_telluric_corrected.fits
../20150411/HIP_42090_telluric_corrected.fits
../20150411/HIP_62541_telluric_corrected.fits
../20150302/HIP_51362_telluric_corrected.fits
../20150302/HIP_61558_telluric_corrected.fits
../20150302/HIP_36760_telluric_corrected.fits
../20150302/HIP_39847_telluric_corrected.fits
../20150302/HIP_19949_telluric_corrected.fits
../20150302/HIP_46897_telluric_corrected.fits
../20150302/HIP_31278_telluric_corrected.fits
../20150302/HIP_60030_telluric_corrected.fits
../20150302/HIP_26093_telluric_corrected.fits
../20150302/HIP_14862_telluric_corrected.fits
../20150302/HIP_45336_telluric_corrected.fits
../20150302/HIP_8704_telluric_corrected.fits
../20150302/HIP_20380_telluric_corrected.fits
../20150302/HIP_25143_telluric_corrected.fits
../20150302/HIP_62541_telluric_corrected.fits
../20141014/HIP_3881_telluric_corrected.fits
../20141014/HIP_84606_telluric_corrected.fits
../20141014/HIP_5310_telluric_corrected.fits
../20141014/HIP_16210_telluric_corrected.fits
../20141014/HIP_19949_telluric_corrected.fits
../20141014/HIP_111056_telluric_corrected.fits
../20141014/HIP_91118_telluric_corrected.fits
../20141014/HIP_101909_telluric_corrected.fits
../20141014/HIP_9564_telluric_corrected.fits
../20141014/HIP_16244_telluric_corrected.fits
../20141014/HIP_10732_telluric_corrected.fits
../20141014/HIP_101909_B_telluric_corrected.fits
../20141014/HIP_109831_telluric_corrected.fits
../20141014/HIP_13879_telluric_corrected.fits
../20141014/HIP_14862_telluric_corrected.fits
../20141014/HIP_99742_telluric_corrected.fits
../20141014/HIP_89935_telluric_corrected.fits
../20141014/HIP_109056_telluric_corrected.fits
../20141014/HIP_108339_telluric_corrected.fits
../20141014/HIP_106786_telluric_corrected.fits
../20141014/HIP_118243_telluric_corrected.fits
../20141014/HIP_16599_telluric_corrected.fits
../20141014/HIP_5626_telluric_corrected.fits
../20141014/HIP_3478_telluric_corrected.fits
../20141014/HIP_2505_telluric_corrected.fits
../20141014/HIP_20380_telluric_corrected.fits
../20141014/HIP_109521_telluric_corrected.fits
../20141014/HR_545_telluric_corrected.fits
../20141014/HIP_101123_telluric_corrected.fits
../20141014/HIP_5518_telluric_corrected.fits
../20141014/HIP_90762_telluric_corrected.fits
../20141014/HIP_104105_telluric_corrected.fits
../20150802/HIP_117089_telluric_corrected.fits
../20150802/HIP_109831_telluric_corrected.fits
../20150802/HIP_104365_telluric_corrected.fits
../20150802/HIP_8704_telluric_corrected.fits
../20150802/HIP_116247_telluric_corrected.fits
../20150726/HIP_101909_telluric_corrected.fits
../20150726/HIP_88818_telluric_corrected.fits
../20150726/HIP_100907_telluric_corrected.fits
../20150726/HIP_115115_telluric_corrected.fits
../20150726/HIP_85998_telluric_corrected.fits
../20150726/HIP_100221_telluric_corrected.fits
../20150726/HIP_103298_telluric_corrected.fits
../20150301/HIP_25813_telluric_corrected.fits
../20150301/HIP_5131_telluric_corrected.fits
../20150301/HIP_10732_telluric_corrected.fits
../20150805/HIP_2548_telluric_corrected.fits
../20150805/HIP_12803_telluric_corrected.fits
../20141017/HD_23872_telluric_corrected.fits
../20141017/HIP_3881_telluric_corrected.fits
../20141017/HIP_36812_telluric_corrected.fits
../20141017/HD_23791_telluric_corrected.fits
../20141017/HD_23629_telluric_corrected.fits
../20141017/HIP_26126_telluric_corrected.fits
../20141017/HIP_84606_telluric_corrected.fits
../20141017/HIP_22833_telluric_corrected.fits
../20141017/HIP_5310_telluric_corrected.fits
../20141017/HIP_16210_telluric_corrected.fits
../20141017/HIP_19949_telluric_corrected.fits
../20141017/HIP_30666_telluric_corrected.fits
../20141017/HIP_17862_telluric_corrected.fits
../20141017/HIP_111056_telluric_corrected.fits
../20141017/HIP_17851_telluric_corrected.fits
../20141017/HD_23733_telluric_corrected.fits
../20141017/HIP_40881_telluric_corrected.fits
../20141017/HIP_24902_telluric_corrected.fits
../20141017/HIP_109056_5_telluric_corrected.fits
../20141017/HIP_91118_telluric_corrected.fits
../20141017/HD_23643_telluric_corrected.fits
../20141017/HIP_33372_telluric_corrected.fits
../20141017/HIP_23375_telluric_corrected.fits
../20141017/HIP_101909_telluric_corrected.fits
../20141017/HIP_109056_8_telluric_corrected.fits
../20141017/HIP_14764_telluric_corrected.fits
../20141017/HIP_20542_telluric_corrected.fits
../20141017/HIP_109056_1_telluric_corrected.fits
../20141017/HIP_93225_telluric_corrected.fits
../20141017/HIP_16611_telluric_corrected.fits
../20141017/HIP_9564_telluric_corrected.fits
../20141017/HIP_29735_telluric_corrected.fits
../20141017/HIP_17791_telluric_corrected.fits
../20141017/HIP_17692_telluric_corrected.fits
../20141017/HD_23863_telluric_corrected.fits
../20141017/HIP_109056_4_telluric_corrected.fits
../20141017/HIP_109056_6_telluric_corrected.fits
../20141017/HIP_85290_telluric_corrected.fits
../20141017/HIP_16244_telluric_corrected.fits
../20141017/HIP_17664_telluric_corrected.fits
../20141017/HIP_22028_telluric_corrected.fits
../20141017/HIP_31278_telluric_corrected.fits
../20141017/HIP_10732_telluric_corrected.fits
../20141017/HIP_20711_telluric_corrected.fits
../20141017/HIP_17999_telluric_corrected.fits
../20141017/HD_23886_telluric_corrected.fits
../20141017/HIP_21670_telluric_corrected.fits
../20141017/HIP_25280_telluric_corrected.fits
../20141017/HIP_21029_telluric_corrected.fits
../20141017/HIP_109831_telluric_corrected.fits
../20141017/HIP_100907_telluric_corrected.fits
../20141017/HIP_26093_telluric_corrected.fits
../20141017/HIP_20789_telluric_corrected.fits
../20141017/HIP_21589_telluric_corrected.fits
../20141017/HIP_44307_telluric_corrected.fits
../20141017/ADS_3962_AB_telluric_corrected.fits
../20141017/HIP_13879_telluric_corrected.fits
../20141017/HIP_14862_telluric_corrected.fits
../20141017/HIP_99742_telluric_corrected.fits
../20141017/HIP_89935_telluric_corrected.fits
../20141017/HD_23479_telluric_corrected.fits
../20141017/HIP_109056_telluric_corrected.fits
../20141017/HD_23924_telluric_corrected.fits
../20141017/HIP_12803_telluric_corrected.fits
../20141017/HIP_108339_telluric_corrected.fits
../20141017/HIP_106786_telluric_corrected.fits
../20141017/HIP_23916_telluric_corrected.fits
../20141017/HIP_16322_telluric_corrected.fits
../20141017/HIP_17776_telluric_corrected.fits
../20141017/HIP_118243_telluric_corrected.fits
../20141017/HIP_16599_telluric_corrected.fits
../20141017/HIP_109056_2_telluric_corrected.fits
../20141017/HIP_5626_telluric_corrected.fits
../20141017/HIP_116611_telluric_corrected.fits
../20141017/HIP_23629_telluric_corrected.fits
../20141017/HIP_20648_telluric_corrected.fits
../20141017/HIP_109056_7_telluric_corrected.fits
../20141017/HD_23567_telluric_corrected.fits
../20141017/HIP_116631_telluric_corrected.fits
../20141017/HIP_3478_telluric_corrected.fits
../20141017/HIP_2505_telluric_corrected.fits
../20141017/HIP_20380_telluric_corrected.fits
../20141017/HIP_12332_telluric_corrected.fits
../20141017/HD_23489_telluric_corrected.fits
../20141017/HIP_17527_telluric_corrected.fits
../20141017/HIP_17579_telluric_corrected.fits
../20141017/HIP_25790_telluric_corrected.fits
../20141017/HIP_109521_telluric_corrected.fits
../20141017/HR_545_telluric_corrected.fits
../20141017/HIP_17588_telluric_corrected.fits
../20141017/HIP_12706_telluric_corrected.fits
../20141017/HIP_101123_telluric_corrected.fits
../20141017/HIP_27713_telluric_corrected.fits
../20141017/HIP_21683_telluric_corrected.fits
../20141017/HIP_15110_telluric_corrected.fits
../20141017/HIP_23362_telluric_corrected.fits
../20141017/HIP_17531_telluric_corrected.fits
../20141017/HIP_5518_telluric_corrected.fits
../20141017/HIP_17608_telluric_corrected.fits
../20141017/HIP_105891_telluric_corrected.fits
../20141017/HIP_29151_telluric_corrected.fits
../20141017/HIP_90762_telluric_corrected.fits
../20141017/HIP_104105_telluric_corrected.fits
../20141017/HIP_109056_3_telluric_corrected.fits
../20141017/HIP_103298_telluric_corrected.fits
../20141017/HIP_25143_telluric_corrected.fits
../20140713/HIP_93805_telluric_corrected.fits
../20141016/HIP_36812_telluric_corrected.fits
../20141016/HIP_40881_telluric_corrected.fits
../20141016/HIP_109056_5_telluric_corrected.fits
../20141016/HIP_33372_telluric_corrected.fits
../20141016/HIP_23375_telluric_corrected.fits
../20141016/HIP_20542_telluric_corrected.fits
../20141016/HIP_109056_1_telluric_corrected.fits
../20141016/HIP_109056_4_telluric_corrected.fits
../20141016/HIP_109056_6_telluric_corrected.fits
../20141016/HIP_17664_telluric_corrected.fits
../20141016/HIP_22028_telluric_corrected.fits
../20141016/HIP_20711_telluric_corrected.fits
../20141016/HIP_25280_telluric_corrected.fits
../20141016/HIP_44307_telluric_corrected.fits
../20141016/HD_23479_telluric_corrected.fits
../20141016/HIP_109056_telluric_corrected.fits
../20141016/HIP_12803_telluric_corrected.fits
../20141016/HIP_109056_2_telluric_corrected.fits
../20141016/HIP_20648_telluric_corrected.fits
../20141016/HIP_109056_7_telluric_corrected.fits
../20141016/HIP_12332_telluric_corrected.fits
../20141016/HD_23489_telluric_corrected.fits
../20141016/HIP_17527_telluric_corrected.fits
../20141016/HIP_17579_telluric_corrected.fits
../20141016/HIP_17588_telluric_corrected.fits
../20141016/HIP_17531_telluric_corrected.fits
../20141016/HIP_17608_telluric_corrected.fits
../20141016/HIP_109056_3_telluric_corrected.fits
../20150409/HIP_62576_telluric_corrected.fits
../20150409/HIP_81126_telluric_corrected.fits
../20150409/HIP_80460_telluric_corrected.fits
../20150409/HIP_88818_telluric_corrected.fits
../20150409/HIP_45336_telluric_corrected.fits
../20150409/HIP_77336_telluric_corrected.fits
../20150409/HIP_85998_telluric_corrected.fits
../20150409/HIP_88817_telluric_corrected.fits
../20150426/HIP_68092_telluric_corrected.fits
../20150426/HIP_61558_telluric_corrected.fits
../20150426/HIP_67143_telluric_corrected.fits
../20150426/HIP_46897_1_telluric_corrected.fits
../20150426/HIP_78820_telluric_corrected.fits
../20150426/HIP95619_telluric_corrected.fits
../20150426/HIP93805_telluric_corrected.fits
../20150426/HIP85922_telluric_corrected.fits
../20150426/HIP_72378_telluric_corrected.fits
../20150426/HIP_78821_telluric_corrected.fits
../20140708/HIP_5131_telluric_corrected.fits
../20140708/HIP_5132_telluric_corrected.fits
../20140708/HIP_104365_telluric_corrected.fits
../20140708/HIP_85922_telluric_corrected.fits
../20140708/HIP_87108_telluric_corrected.fits
../20140708/HIP_102487_telluric_corrected.fits
../20140708/HR_6025_telluric_corrected.fits

In [3]:
# Guess stellar properties
MS = SpectralTypeRelations.MainSequence()
def guess_teff_logg(fname):
    header = fits.getheader(fname)
    data = StarData.GetData(header['OBJECT'])
    spt = data.spectype
    teff = MS.Interpolate('Temperature', spt)
    logg = 3.5 if 'I' in spt else 4.0
    return teff, logg

In [4]:
teff, logg = guess_teff_logg(star_list[0])
print(teff, logg)


INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): simbak.cfa.harvard.edu
(array(9700.0), 4.0)

In [6]:
# Read in flat lamp spectrum (it is not flat!)
from scipy.interpolate import InterpolatedUnivariateSpline as spline
import numpy as np
out = np.loadtxt('../plp/flat_lamp.txt', unpack=True)
wvl, fl = out[:, 0], out[:, 1]
flat = spline(wvl, fl)

In [7]:
import HelperFunctions
orders = HelperFunctions.ReadExtensionFits(star_list[2])

import matplotlib.pyplot as plt
%matplotlib notebook
#nums = tuple(range(5, 16)) + tuple(range(18, 26))
nums = range(3, 18)
#for order in orders[4:25]:
n_left, n_right = 250, 100
good_orders = [o[n_left:-n_right].copy() for i, o in enumerate(orders) if i in nums]
for order in good_orders:
    plt.plot(order.x, order.y, 'k-', alpha=0.5)
    plt.plot(order.x, order.y*flat(order.x), 'r-', alpha=0.5)



In [9]:
reload(SpecFlattener)
print(len(nums))
output = SpecFlattener.flatten_spec(star_list[2], hdf5_lib, teff=teff, logg=logg, normalize_model=False,
                                    ordernums=nums, x_degree=4, orders=good_orders)
final_orders, flattened, shifted_orders, mcf = output


INFO:root:Did not find file (../20140709/HIP_85537_telluric_corrected.fits) in log (Flatten.log)
15
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-9-c42b44ba39be> in <module>()
      2 print(len(nums))
      3 output = SpecFlattener.flatten_spec(star_list[2], hdf5_lib, teff=teff, logg=logg, normalize_model=False,
----> 4                                     ordernums=nums, x_degree=4, orders=good_orders)
      5 final_orders, flattened, shifted_orders, mcf = output

/home/kgullikson/.PythonModules/GeneralScripts/SpecFlattener.py in flatten_spec(filename, hdf5_lib, teff, logg, feh, orders, first_order, last_order, ordernums, x_degree, y_degree, normalize_model, summary_file)
    457 
    458     mcf = ModelContinuumFitter(orders, hdf5_lib, x_degree=x_degree, y_degree=y_degree,
--> 459                                T=teff, logg=logg, feh=feh, initialize=True, order_numbers=ordernums)
    460     logging.info('RV guess = {}\n\tvsini guess = {}'.format(mcf.rv_guess, mcf.vsini_guess))
    461 

/home/kgullikson/.PythonModules/GeneralScripts/SpecFlattener.py in __init__(self, data_orders, model_library, x_degree, y_degree, wave_spacing, T, logg, feh, initialize, order_numbers, **kwargs)
    130         self._logg = None
    131         self._feh = None
--> 132         hdf5_int = StellarModel.HDF5Interface(model_library)
    133         dataspec = StellarModel.DataSpectrum(wls=wl, fls=fl, sigmas=sig)
    134         self.interpolator = StellarModel.Interpolator(hdf5_int, dataspec)

/home/kgullikson/.PythonModules/GeneralScripts/StellarModel.pyc in __init__(self, filename, ranges)
    312         grid_set = frozenset(grid_parameters)
    313 
--> 314         with h5py.File(self.filename, "r") as hdf5:
    315             self.wl = hdf5["wl"][:]
    316             self.wl_header = dict(hdf5["wl"].attrs.items())

/home/kgullikson/anaconda3/envs/python2/lib/python2.7/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, driver, libver, userblock_size, swmr, **kwds)
    258 
    259                 fapl = make_fapl(driver, libver, **kwds)
--> 260                 fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
    261 
    262                 if swmr_support:

/home/kgullikson/anaconda3/envs/python2/lib/python2.7/site-packages/h5py/_hl/files.pyc in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
     87         if swmr and swmr_support:
     88             flags |= h5f.ACC_SWMR_READ
---> 89         fid = h5f.open(name, flags, fapl=fapl)
     90     elif mode == 'r+':
     91         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.5.0/h5py/_objects.c:2458)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py-2.5.0/h5py/_objects.c:2415)()

h5py/h5f.pyx in h5py.h5f.open (-------src-dir-------/h5py-2.5.0/h5py/h5f.c:1792)()

IOError: Unable to open file (Unable to open file: name = '/volumes/datadrive/kurucz_grid/igrins_grid_full.hdf5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)

In [42]:
%matplotlib notebook
for order in final_orders:
    plt.plot(order.x, order.y, 'k-', alpha=0.5)



In [ ]: