SPIRE FTS data access and data analysis in python 3

Introduction

The Herschel Science Archive has a I/O interface called HAIO. How to use it is explained in some details in this url http://archives.esac.esa.int/hsa/aio/doc/howto.html

In order to access Herschel data, one can construct an HAIO HTTP request and submit it to the archive.

In the following example, we download SPIRE FTS level-2 data as a tar file, extract the relevant fits file and read it. Then we reorganise the spectral data in a dictionary.


In [1]:
#
# import some necessary packages
#
import matplotlib.pyplot as plt
%matplotlib inline
from astropy.io import fits
import tarfile
import requests
import tempfile
import os, shutil

In [2]:
def getSpireFtsLevel2(obsid, what='spss', archive='hsa', saveTar=True):
    """
    PURPOSE:
        Using the HTTP access to HAIO, retrieve all level-2 products in a tar file
        and extract only the requested fits file (spss in the name) at then put the data in 
        a dictionary
    INPUTS:
        obsid - the OBSID of the observation
        what - can be 'spss' for point source calibrated spectra or 'sds' for extended source 
        calibrated data. Note that only non-apodized spectra are searched for. You can change this 
        behaviour in the function below, by replacing '_spg_' with '_spgApod_'.
        archive - can be 'hsa' or 'hsaint' (for the integration archive).
        saveTar - whether to save the downloaded level-2 tar file in the current folder. The tar file name 
            will be OBSID_level2.tar. If False, the file will be deleted after the spss FITS file is extracted.
    OUTPUTS:
        A nested dictionary with the following structure:
            spec[det]['wave'], 
            spec[det]['flux'], 
            spec[det]['fluxErr'], 
            where det is the name of the detector.
        The FITS header of the primary extension of the fits file.
    EXAMPLE:
        spss, header = getSpireFtsLevel2(1342259588)
        spss.keys()
        spss["SSWD4"].keys()
        header
    TODO:
        * Decide what to extract from the tar file when the observation is in H+LR mode.
    HISTORY:
        Created 16 Mar 2016 (IV), version 1.0
    """
    # the name of the saved tar file.
    tarFile = "%i_level2.tar"%obsid
    archiveUrl = "http://archives.esac.esa.int/%s/aio/jsp/product.jsp"%archive
    haioRequest = "%s?PROTOCOL=HTTP&OBSERVATION_ID=%i&PRODUCT_LEVEL=Level2"%(archiveUrl,obsid)
    print ("Downloading level-2 data from the Herschel Science Archive. May take a while... be patient")
    #
    # submit the HAIO request to the server
    #
    r = requests.get(haioRequest)
    #
    # save the result in a .tar file
    #
    with open(tarFile, "wb") as tmp:
        tmp.write(r.content)
    #
    # now read the downloaded tar file
    #
    xx = None
    try:
        tar = tarfile.open(tarFile,'r')
        for member in tar.getnames():
            tmpdir = tempfile.mkdtemp()
            if ((what in member) and ('_spg_' in member)):
                f=tar.extract(member,tmpdir)
                xx = fits.open(os.path.join(tmpdir,member))
                break
            # clean up the temprary folder with the extracted file
            shutil.rmtree(tmpdir)
            # remove the tar file if asked for.
            if (not saveTar): os.remove(tarFile)
    except:
        print ("*** OBSID %i is not available or it is proprietory."%obsid)
        return None, None
    finally:
        tar.close()
    if (xx == None):
        print ("*** OBSID %i is not an FTS sparse mode. Return."%obsid)
        return None, None
    spec = {}
    # get the primary header
    header = xx[0].header
    for k in xx:
        extname = k.name
        if ('S' in extname):
            spec[k.name] = {}
            spec[k.name]['wave'] = k.data["wave"]
            spec[k.name]['flux'] = k.data["flux"]
            spec[k.name]['fluxErr'] = k.data["error"]
    return spec, header

Now, let's try it with one observation.


In [3]:
# try a proprietory data
#spss2, hh2 = getSpireFtsLevel2(1342246269)
#
# now a normal one
spss, hh = getSpireFtsLevel2(1342259588)
print (spss.keys())
print (spss["SSWD4"].keys())
hh
# now not a spec one
#spss2, hh2 = getSpireFtsLevel2(1342195774)


Downloading level-2 data from the Herschel Science Archive. May take a while... be patient
dict_keys(['SLWB2', 'SSWF2', 'SLWB3', 'SSWD2', 'SSWC2', 'SLWC3', 'SSWF3', 'SSWB4', 'SSWC4', 'SSWE3', 'SLWD2', 'SSWE5', 'SSWC5', 'SSWB2', 'SLWC4', 'SLWD3', 'SSWD3', 'SSWD4', 'SSWD6', 'SSWC3', 'SLWC2', 'SSWB3', 'SSWE4', 'SSWE2'])
dict_keys(['fluxErr', 'flux', 'wave'])
Out[3]:
SIMPLE  =                    T / Java FITS: Tue Feb 09 09:41:16 CET 2016        
BITPIX  =                   32                                                  
NAXIS   =                    0 / Dimensionality                                 
EXTEND  =                    T / May contain datasets                           
TIMESYS = 'UTC     '           / All dates are in UTC time                      
LONGSTRN= 'OGIP 1.0'           / The OGIP long string convention may be used.   
COMMENT This FITS file may contain long string keyword values that are          
COMMENT continued over multiple keywords.  This convention uses the  '&'        
COMMENT character at the end of a string which is then continued                
COMMENT on subsequent keywords whose name = 'CONTINUE'.                         
          ---------------Herschel FITS Data Generator---------------            
          This product is generated by Herschel software.                       
HCSS____=                    5 / HCSS Fits Product Version                      
          -------------- Herschel Structure Data--------------------            
          Following fields are private to the structure of the                  
          Java object this HDU is representing.                                 
CLASS___= 'herschel.spire.ia.dataset.SpectrometerPointSourceSpectrum&'          
CONTINUE  '' / &                                                                
COMMENT java representation                                                     
INFO____= 'Spectrometer Point Source Spectrum'                                  
          -------------- Herschel Parameter Data--------------------            
          All actual parameter names are converted to FITS compliant            
          conventions. Note that the HIERARCH comments contain the              
          appropriate key mapping                                               
TELESCOP= 'Herschel Space Observatory' / Name of telescope                      
INSTRUME= 'SPIRE   '           / Instrument attached to this product            
SUBSYS  = 'SPECTROMETER'       / Instrument Subsystem                           
SOURCE  = 'SPECF   '           / TM source packet name                          
CREATOR = 'SPG v14.0.1'        / Generator of this product                      
OBJECT  = 'Uranus  '           / Target name                                    
OBSERVER= 'rpspire '           / Observer name                                  
PROPOSAL= 'Calibration_rpspire_152' / Proposal name                             
OBS_ID  =           1342259588 / [] Observation identifier                      
ODNUMBER=                 1341 / [] Operational day number                      
DATE    = '2016-02-09T08:48:31.960000' / Creation date of this product          
DATE-OBS= '2013-01-14T08:00:51.855206' / Start date of this product             
DATE_OBS= '2013-01-14T08:00:51.855206' / Start date of this product             
DATE-END= '2013-01-14T08:09:54.599729' / End date of this product               
AOT     = 'Spectrometer'       / AOT Identifier                                 
AOR     = 'Calibration_cycle83_1-SpireSpectroPointGen-HR-Rep4-Uranus&'          
CONTINUE  '' / &                                                                
COMMENT AOR Label as entered in HSpot                                           
CUSMODE = 'SpireSpectroPoint'  / CUS observation mode                           
INSTMODE= 'SOF1    '           / Instrument Mode                                
OBS_MODE= 'Single Pointing'    / Observation mode name                          
POINTMOD= 'Basic-fine'         / Pointing mode                                  
MAPSAMPL= 'sparse  '           / Spatial sampling of map                        
BIASMODE= 'nominal '           / Bias mode                                      
JIGG_ID =                    0 / [] Jiggle ID                                   
POINT_ID=                    0 / [] Pointing number                             
NUMREP  =                    4 / [] Number of times to repeat the basic unit &  
COMMENT of the observation                                                      
ONSRCTIM=    530.0760803222656 / [s] Total on-source integration time           
CMDRES  = 'HR      '           / Commanded Spectral Resolution                  
BIASFREQ=    160.0922131147541 / [Hz] Bias frequency                            
ELECSIDE= 'prime   '           / Electronic side                                
MODELNAM= 'FLIGHT  '           / Model name attached to this product            
FORMATV = '1.0     '           / Version of product format                      
MISSIONC= 'MC_H102ASTR_P70ASTR_S66ASTR_RP' / Mission configuration              
EQUINOX =               2000.0 / [] Equinox of celestial coordinate system      
RADESYS = 'ICRS    '           / Coordinate reference frame for the RA and DEC  
RA      =    4.713194358720781 / [deg] Actual Right Ascension of pointing       
RA_NOM  =    4.713035338744305 / [deg] Requested Right Ascension of pointing    
DEC     =   1.2815238756437137 / [deg] Actual Declination of pointing           
DEC_NOM =    1.281942219956127 / [deg] Requested Declination of pointing        
POSANGLE=   246.95669072167988 / [deg] Position Angle of pointing               
PMRA    =                  0.0 / [arcsec a-1] Target's proper motion RA (arcs&  
COMMENT ec/yr) as given by the observer                                         
PMDEC   =                  0.0 / [arcsec a-1] Target's proper motion Dec (arc&  
COMMENT sec/yr) as given by the observer                                        
RADECOFF=    1.611122331062671 / [arcsec] The offset between the commanded po&  
COMMENT sition and the actual reconstructed pointing, which includes any system&
COMMENT atic BSM offset (bsmOffset), but not the APE                            
BSMOFF  =                  0.0 / [arcsec] BSM offset position (0.0 or 1.7 arc&  
COMMENT sec), which is corrected for by the point-source calibration. This syst&
COMMENT ematic offset is included in the raDecOffset value                      
VFRAME  =    -30.0061819278842 / [km s-1] Spacecraft velocity along the l-of-&  
COMMENT s of the telescope wrt the LSR                                          
VELDEF  = 'RADI-LSR'           / The velocity definition and frame              
ORIGIN  = 'Herschel Science Centre' / Site that created the product             
PROCMODE= 'BULK_REPROCESSING&'                                                  
CONTINUE  '' / &                                                                
COMMENT Processing mode selected to execute the pipeline                        
TYPE    = 'SPSS    '           / Product Type Identification                    
DESC    = 'Spectrometer Point Source Spectrum' / Name of this product           
ECALVERS= 'spire_cal_14_2&'                                                     
CONTINUE  '' / &                                                                
COMMENT Version of the calibration tree used in the engineering conversion      
CALVERS = 'spire_cal_14_2'     / Calibration version                            
LEVEL   = '20      '           / The level of the product                       
PROC_RES= 'HR      '           / The resolution used to process the data        
NUMSCANS=                    1 / [] Number of Scans                             
BBID    =           2701524993 / [] Building Block Identifier                   
BBTNAME = 'SpireBbFtsScan'     / Building block type name                       
SPECSYS = 'LSRk    '           / Standard of rest for spectral axis             
APODTYPE= 'None    '           / Type of Apodization applied                    
APODNAME= 'unapod  '           / Name of Apodization function applied           
M1TEMP  =    89.63837490312442 / [K] Telescope temperature M1 (in K)            
M2TEMP  =    85.12922833897763 / [K] Telescope temperature M2 (in K)            
PCAPPL  =                    T / Phase correction has been applied              
SIGTABNM= 'voltage '           / Name of the signal table                       
MSKDEAD =                  128 / [] Mask value for dead channel                 
MSKMSTR =                    1 / [] Mask value for master bit                   
MSKNOISY=                  256 / [] Mask value for noisy channel                
MSKSLOW =                32768 / [] Mask value for slow channel                 
ADCERFLG=                    F / Flag for presence of ADC Latch errors          
RATOORSS= 2.497353065791685E-4 / [] Largest ratio of number of out-of-range s&  
COMMENT amples to total number of samples in SSW data                           
RATOORSL= 1.639966867514886E-4 / [] Largest ratio of number of out-of-range s&  
COMMENT amples to total number of samples in SLW data                           
MSKADC  =                    4 / [] Mask value for possible ADC latchup error   
MSKTRUNC=                    8 / [] Mask value for ADC conversion truncation    
MSKUNCTR=                   16 / [] Mask value for uncorrected ADC conversion&  
COMMENT ~truncation                                                             
RINVTIME=                  0.0 / [] Fraction of invalid sample times            
MSKINVTM=                    2 / [] Mask value for invalid sample time          
TIMEOFST=                0.079 / [s] Time offset added to correct the shift b&  
COMMENT etween detectors and pointing timing                                    
TIMEDRFT=                -0.39 / [ms/min] Time drift added to correct the shi&  
COMMENT ft between detectors and pointing timing                                
INVOFFLG=                    F / If true, offsets are from a previous observa&  
COMMENT tion                                                                    
SLWBIASA=          0.031132395 / [V] SLW bias amplitude                         
SSWBIASA=  0.03596871200000001 / [V] SSW bias amplitude                         
SLWBIASP= -0.18480026085660228 / [rad] SLW bias phase                           
SSWBIASP=  -0.1108804183133494 / [rad] SSW bias phase                           
RCRLCORR=                    F / RC roll correction applied                     
SUBKDRFT=    6.535649299644E-6 / [K] Drift measured by the SubK temperature p&  
COMMENT robe                                                                    
MSKGLTC1=                 2048 / [] Mask value for first level glitch detected  
MSKGNR1 =                 4096 / [] Mask value for first level glitch detecte&  
COMMENT d and not removed                                                       
QCR1GSL =  0.00679579247801869 / [] Maximum ratio of number of flagged sample&  
COMMENT s over number of data samples in the detectors of the SLW array         
QCR1GSS = 0.006821686691525112 / [] Maximum ratio of number of flagged sample&  
COMMENT s over number of data samples in the detectors of the SSW array         
QCROCRSS= 4.406126488711767E-5 / [] Ratio of number of samples that are out o&  
COMMENT f the calibrated voltage range over number of data samples in the detec&
COMMENT tors of this array                                                      
NBK3VSS =                    0 / [] Number of samples measured by any detecto&  
COMMENT r in this array for which the voltage is less than the value of the K3 &
COMMENT calibration parameter                                                   
QCROCRSL=                  0.0 / [] Ratio of number of samples that are out o&  
COMMENT f the calibrated voltage range over number of data samples in the detec&
COMMENT tors of this array                                                      
NBK3VSL =                    0 / [] Number of samples measured by any detecto&  
COMMENT r in this array for which the voltage is less than the value of the K3 &
COMMENT calibration parameter                                                   
MSKVOOL =                 1024 / [] Mask value for voltage out of fitted range  
MSKBEL3K=                65536 / [] Mask value for voltage below K3             
MSKNORSP=               131072 / [] Mask value for voltage out of fitted rang&  
COMMENT e and/or below K3                                                       
SPCNLCOR=                    T / Non-linearity correction has been applied      
RESPCTRL= '2014-02-13T08:00:00.000000&'                                         
CONTINUE  '' / &                                                                
COMMENT Control Stamp used to ensure synchronization of Flux Conversion and Tem&
COMMENT perature Drift correction tasks                                         
NCLPNCSL=                    0 / [] Maximum number of detectors in SLW where &  
COMMENT at least one sample was flagged as clipped and could not be corrected   
NCLPNCSS=                    0 / [] Maximum number of detectors in SSW where &  
COMMENT at least one sample was flagged as clipped and could not be corrected   
SCNEXTRM=   1.0004079040404041 / [] Maximum ratio of mesured SMEC scan extrem&  
COMMENT a to commanded scan extrema                                             
SMECTEMP=  0.05615854548081689 / [] Maximum ddifference between maximum and m&  
COMMENT inimum SMEC temperatures (in K)                                         
QCMSFRNG=                    0 / [] Maximum number of missed optical encoder &  
COMMENT fringes                                                                 
SSPDAVG =   0.9956572934492893 / [] Maximum ratio of the measured SMEC speed &  
COMMENT to the commanded SMEC speed                                             
SSPDSDEV=   17.416762237954735 / [] Maximum standard deviation of the SMEC sp&  
COMMENT eed (in micron/s)                                                       
QCSMCOLC=                    0 / [] Number of housekeeping entries which indi&  
COMMENT cate that the SMEC is not in closed loop                                
QCWGBLSS=                    0 / [] Maximum number of detectors in SSW where &  
COMMENT the removed baseline is not a good approximation of the actual baseline&
COMMENT ~in the interferogram                                                   
QCWGBLSL=                    0 / [] Maximum number of detectors in SLW where &  
COMMENT the removed baseline is not a good approximation of the actual baseline&
COMMENT ~in the interferogram                                                   
MSKGLTCH=                   32 / [] Mask value for glitch detected              
MSKGNR  =                   64 / [] Mask value for glitch detected and not re&  
COMMENT moved                                                                   
MSKUNREL=                  128 / [] Mask value for unreliable telemetry param&  
COMMENT eter                                                                    
MSKNCTOS=                  512 / [] Mask value for channel not chopped to sky   
MSKGLTC2=                 8192 / [] Mask value for second level glitch detect&  
COMMENT ed                                                                      
MSKGNR2 =                16384 / [] Mask value for second level glitch detect&  
COMMENT ed and not removed                                                      
MSKTSIG =               262144 / [] Mask value for thermistor/DP signal devia&  
COMMENT tions are larger than expected                                          
MSKCOOL =               524288 / [] Mask value for sample falls outside BSM c&  
COMMENT hop soft limits                                                         
MSKJOOL =              1048576 / [] Mask value for sample falls outside BSM J&  
COMMENT iggle soft limits                                                       
MSKJUMP =              2097152 / [] Mask value for signal jump in the thermis&  
COMMENT tor/dark timeline                                                       
MSKNOTHM=              4194304 / [] Mask value for all the available thermist&  
COMMENT ors are affected by either saturation or signal jump                    
MSKNOMVL=              8388608 / [] Mask value for sample is not part of the &  
COMMENT nominal science scans of a scan map                                     
MSKDKCHN=             16777216 / [] Mask value for all available dark channel&  
COMMENT s in the array are affected by either saturation or signal jump         
MSKSNDIP=             67108864 / [] Mask value for sample belongs to serendip&  
COMMENT ity mode                                                                
MSKTRNRD=             33554432 / [] Mask value for sample belongs to a turnar&  
COMMENT ound section of a scan map                                              
QCR2GSS = 8.207463049528493E-4 / [] Maximum ratio of number of flagged sample&  
COMMENT s over number of data samples in the detectors of the SSW array         
QCR2GSL = 9.170955571262182E-4 / [] Maximum ratio of number of flagged sample&  
COMMENT s over number of data samples in the detectors of the SLW array         
QCR2GPSS=                  0.0 / [] Maximum ratio of number of data samples w&  
COMMENT here there were not enough valid scans to perform statistical outlier d&
COMMENT etection over the total number of data samples in the detectors of the &
COMMENT SSW array                                                               
QCR2GPSL=                  0.0 / [] Maximum ratio of number of data samples w&  
COMMENT here there were not enough valid scans to perform statistical outlier d&
COMMENT etection over the total number of data samples in the detectors of the &
COMMENT SLW array                                                               
NL2GSS  =                    0 / [] Maximum number of cases where a data samp&  
COMMENT le from a SSW detector that was flagged as a glitch was not corrected   
NL2GSL  =                    0 / [] Maximum number of cases where a data samp&  
COMMENT le from a SLW detector that was flagged as a glitch was not corrected   
PCWRAPSL=                    0 / [] Maximum number of times that in-band phas&  
COMMENT e changes by +/-2pi in any of the SLW detectors                         
PCWRAPSS=                    0 / [] Maximum number of times that in-band phas&  
COMMENT e changes by +/-2pi in any of the SSW detectors                         
QCRSPHSL=    5.154865454716195 / [] Maximum phase within the phase correction&  
COMMENT ~limits (in units of degree) for SLW                                    
QCRSPHSW=   1.2636185396979518 / [] Maximum phase within the phase correction&  
COMMENT ~limits (in units of degree) for SSW                                    
RATOBHSL=    0.002461657903578 / [] Maximum ratio of the mean of the absolute&  
COMMENT ~flux between 1200 and 5996 GHz over the mean of the absolute flux with&
COMMENT in the band edges.                                                      
RATOBHSS=    0.005543089344431 / [] Maximum ratio of the mean of the absolute&  
COMMENT ~flux between 1799 and 5996 GHz over the mean of the absolute flux with&
COMMENT in the band edges.                                                      
RATOBLSL= 0.003829212937135345 / [] Maximum ratio of the mean of the absolute&  
COMMENT ~flux between 0.03 and 360 GHz over the mean of the absolute flux withi&
COMMENT n the band edges.                                                       
RATOBLSS= 0.007274190155717046 / [] Maximum ratio of the mean of the absolute&  
COMMENT ~flux between 0.03 and 900 GHz over the mean of the absolute flux withi&
COMMENT n the band edges.                                                       
DSTEMP  = 0.030429037788570312 / [K] SCal maximum temperature variation (in K)  
DS2TEMP = 0.024681563997869915 / [K] SCal2 maximum temperature variation (in &  
COMMENT K)                                                                      
DS4TEMP = 0.022776108255117222 / [K] SCal4 maximum temperature variation (in &  
COMMENT K)                                                                      
DM1TEMP =                  0.0 / [K] Maximum temperature variation of the tel&  
COMMENT escope 1 temperature (in K)                                             
DM2TEMP =                  0.0 / [K] Maximum temperature variation of the tel&  
COMMENT escope 2 temperature (in K)                                             
SLEWTIME= '2013-01-14T08:00:11.000000' / Scheduled start time of the slew       
FILENAME= 'hspirespectrometer1342259588_a1060001_spg_HR_20spss'                 
HIERARCH  key.TELESCOP='telescope'                                              
HIERARCH  key.INSTRUME='instrument'                                             
HIERARCH  key.SUBSYS='subsystem'                                                
HIERARCH  key.SOURCE='source'                                                   
HIERARCH  key.CREATOR='creator'                                                 
HIERARCH  key.OBJECT='object'                                                   
HIERARCH  key.OBSERVER='observer'                                               
HIERARCH  key.PROPOSAL='proposal'                                               
HIERARCH  key.OBS_ID='obsid'                                                    
HIERARCH  key.ODNUMBER='odNumber'                                               
HIERARCH  key.DATE='creationDate'                                               
HIERARCH  key.DATE-OBS='startDate'                                              
HIERARCH  key.DATE-END='endDate'                                                
HIERARCH  key.AOT='aot'                                                         
HIERARCH  key.AOR='aorLabel'                                                    
HIERARCH  key.CUSMODE='cusMode'                                                 
HIERARCH  key.INSTMODE='instMode'                                               
HIERARCH  key.OBS_MODE='obsMode'                                                
HIERARCH  key.POINTMOD='pointingMode'                                           
HIERARCH  key.MAPSAMPL='mapSampling'                                            
HIERARCH  key.BIASMODE='biasMode'                                               
HIERARCH  key.JIGG_ID='jiggId'                                                  
HIERARCH  key.POINT_ID='pointNum'                                               
HIERARCH  key.NUMREP='numRepetitions'                                           
HIERARCH  key.ONSRCTIM='onSourceTime'                                           
HIERARCH  key.CMDRES='commandedResolution'                                      
HIERARCH  key.BIASFREQ='biasFreq'                                               
HIERARCH  key.ELECSIDE='elecSide'                                               
HIERARCH  key.MODELNAM='modelName'                                              
HIERARCH  key.FORMATV='formatVersion'                                           
HIERARCH  key.MISSIONC='missionConfig'                                          
HIERARCH  key.EQUINOX='equinox'                                                 
HIERARCH  key.RADESYS='raDeSys'                                                 
HIERARCH  key.RA='ra'                                                           
HIERARCH  key.RA_NOM='raNominal'                                                
HIERARCH  key.DEC='dec'                                                         
HIERARCH  key.DEC_NOM='decNominal'                                              
HIERARCH  key.POSANGLE='posAngle'                                               
HIERARCH  key.PMRA='pmRA'                                                       
HIERARCH  key.PMDEC='pmDEC'                                                     
HIERARCH  key.RADECOFF='raDecOffset'                                            
HIERARCH  key.BSMOFF='bsmOffset'                                                
HIERARCH  key.VFRAME='radialVelocity'                                           
HIERARCH  key.VELDEF='velocityDefinition'                                       
HIERARCH  key.ORIGIN='origin'                                                   
HIERARCH  key.PROCMODE='processingMode'                                         
HIERARCH  key.TYPE='type'                                                       
HIERARCH  key.DESC='description'                                                
HIERARCH  key.ECALVERS='engConvCalVersion'                                      
HIERARCH  key.CALVERS='calVersion'                                              
HIERARCH  key.LEVEL='level'                                                     
HIERARCH  key.PROC_RES='processResolution'                                      
HIERARCH  key.NUMSCANS='numScans'                                               
HIERARCH  key.BBID='bbid'                                                       
HIERARCH  key.BBTNAME='bbTypeName'                                              
HIERARCH  key.SPECSYS='freqFrame'                                               
HIERARCH  key.APODTYPE='apodType'                                               
HIERARCH  key.APODNAME='apodName'                                               
HIERARCH  key.M1TEMP='telescopeTempM1'                                          
HIERARCH  key.M2TEMP='telescopeTempM2'                                          
HIERARCH  key.PCAPPL='phaseCorrApplied'                                         
HIERARCH  key.SIGTABNM='signalTable'                                            
HIERARCH  key.MSKDEAD='maskDead'                                                
HIERARCH  key.MSKMSTR='maskMaster'                                              
HIERARCH  key.MSKNOISY='maskNoisy'                                              
HIERARCH  key.MSKSLOW='maskSlow'                                                
HIERARCH  key.ADCERFLG='adcErrFlag'                                             
HIERARCH  key.RATOORSS='ratioTruncatedSSW'                                      
HIERARCH  key.RATOORSL='ratioTruncatedSLW'                                      
HIERARCH  key.MSKADC='maskAdcLatch'                                             
HIERARCH  key.MSKTRUNC='maskTruncated'                                          
HIERARCH  key.MSKUNCTR='maskUncorrectedTruncation'                              
HIERARCH  key.RINVTIME='ratioInvalidTimes'                                      
HIERARCH  key.MSKINVTM='maskInvalidTime'                                        
HIERARCH  key.TIMEOFST='timeOffset'                                             
HIERARCH  key.TIMEDRFT='timeDrift'                                              
HIERARCH  key.INVOFFLG='invalidOffsetFlag'                                      
HIERARCH  key.SLWBIASA='slwBiasAmpl'                                            
HIERARCH  key.SSWBIASA='sswBiasAmpl'                                            
HIERARCH  key.SLWBIASP='slwBiasPhase'                                           
HIERARCH  key.SSWBIASP='sswBiasPhase'                                           
HIERARCH  key.RCRLCORR='rcRollApp'                                              
HIERARCH  key.SUBKDRFT='subkTempDrift'                                          
HIERARCH  key.MSKGLTC1='maskGlitchL1Detected'                                   
HIERARCH  key.MSKGNR1='maskGlitchL1NotRemoved'                                  
HIERARCH  key.QCR1GSL='ratioSpecFirstLevelGlitchesSLW'                          
HIERARCH  key.QCR1GSS='ratioSpecFirstLevelGlitchesSSW'                          
HIERARCH  key.QCROCRSS='ratioSamplesOutOfCalibrationRangeSSW'                   
HIERARCH  key.NBK3VSS='numberBelowK3VoltageSSW'                                 
HIERARCH  key.QCROCRSL='ratioSamplesOutOfCalibrationRangeSLW'                   
HIERARCH  key.NBK3VSL='numberBelowK3VoltageSLW'                                 
HIERARCH  key.MSKVOOL='maskVoltageOol'                                          
HIERARCH  key.MSKBEL3K='maskVoltageBelowK3'                                     
HIERARCH  key.MSKNORSP='maskNoRespData'                                         
HIERARCH  key.SPCNLCOR='SpecNonLinearityCorrectionDone'                         
HIERARCH  key.RESPCTRL='respControlStamp'                                       
HIERARCH  key.NCLPNCSL='numberClippedDetectorsSLW'                              
HIERARCH  key.NCLPNCSS='numberClippedDetectorsSSW'                              
HIERARCH  key.SCNEXTRM='scanExtrema'                                            
HIERARCH  key.SMECTEMP='smecTemperature'                                        
HIERARCH  key.QCMSFRNG='missedFringes'                                          
HIERARCH  key.SSPDAVG='stageSpeedAverage'                                       
HIERARCH  key.SSPDSDEV='stageSpeedDeviation'                                    
HIERARCH  key.QCSMCOLC='smecOpenLoopCount'                                      
HIERARCH  key.QCWGBLSS='wrongBaselineSSW'                                       
HIERARCH  key.QCWGBLSL='wrongBaselineSLW'                                       
HIERARCH  key.MSKGLTCH='maskGlitchDetected'                                     
HIERARCH  key.MSKGNR='maskGlitchNotRemoved'                                     
HIERARCH  key.MSKUNREL='maskUnreliableParameter'                                
HIERARCH  key.MSKNCTOS='maskNotChoppedToSky'                                    
HIERARCH  key.MSKGLTC2='maskGlitchL2Detected'                                   
HIERARCH  key.MSKGNR2='maskGlitchL2NotRemoved'                                  
HIERARCH  key.MSKTSIG='maskTsignalHdv'                                          
HIERARCH  key.MSKCOOL='maskBsmChopOol'                                          
HIERARCH  key.MSKJOOL='maskBsmJiggOol'                                          
HIERARCH  key.MSKJUMP='maskJumpThermistorsDarksSignal'                          
HIERARCH  key.MSKNOTHM='maskNoThermistorAvailable'                              
HIERARCH  key.MSKNOMVL='maskNonNominalVelocity'                                 
HIERARCH  key.MSKDKCHN='maskNoDarkChannelAvailable'                             
HIERARCH  key.MSKSNDIP='maskSerendipity'                                        
HIERARCH  key.MSKTRNRD='maskTurnaround'                                         
HIERARCH  key.QCR2GSS='ratioSpecSecondLevelGlitchesSSW'                         
HIERARCH  key.QCR2GSL='ratioSpecSecondLevelGlitchesSLW'                         
HIERARCH  key.QCR2GPSS='ratioSpecSecondLevelGlitchIdentificationProblemsSSW'    
HIERARCH  key.QCR2GPSL='ratioSpecSecondLevelGlitchIdentificationProblemsSLW'    
HIERARCH  key.NL2GSS='numberSecondLevelGlitchCorrectionProblemsSSW'             
HIERARCH  key.NL2GSL='numberSecondLevelGlitchCorrectionProblemsSLW'             
HIERARCH  key.PCWRAPSL='phaseWrapSLW'                                           
HIERARCH  key.PCWRAPSS='phaseWrapSSW'                                           
HIERARCH  key.QCRSPHSL='residualPhaseSLW'                                       
HIERARCH  key.QCRSPHSW='residualPhaseSSW'                                       
HIERARCH  key.RATOBHSL='outOfBandHighRatioSLW'                                  
HIERARCH  key.RATOBHSS='outOfBandHighRatioSSW'                                  
HIERARCH  key.RATOBLSL='outOfBandLowRatioSLW'                                   
HIERARCH  key.RATOBLSS='outOfBandLowRatioSSW'                                   
HIERARCH  key.DSTEMP='deltaSCalTemp'                                            
HIERARCH  key.DS2TEMP='deltaSCal2Temp'                                          
HIERARCH  key.DS4TEMP='deltaSCal4Temp'                                          
HIERARCH  key.DM1TEMP='deltaM1Temp'                                             
HIERARCH  key.DM2TEMP='deltaM2Temp'                                             
HIERARCH  key.SLEWTIME='slewTime'                                               
HIERARCH  key.FILENAME='fileName'                                               
          -----------------Dataset Children-------------------------            
          Below you will find all HDU locations that are children of            
          this composite dataset.                                               
DSETS___=                    1 / Number of datasets                             
DS_0    =                    1 / HDU of Child Dataset                           

Let's try to plot all spectra, the central detectors with one colour and the off-axis ones in another.


In [4]:
def plotSpireSparseSpec(specIn, fitsHeader=None, onlyCentral=True):
    """
    """
    central = ['SSWD4','SLWC3']
    fig = plt.figure(figsize=(8,5))
    for det in central:
        plt.plot(specIn[det]['wave'],specIn[det]['flux'],'k-')
    if (not onlyCentral): 
        for det in specIn.keys():
            if (not det in central):
                plt.plot(specIn[det]['wave'],specIn[det]['flux'],'c-')
    plt.xlabel('Frequency (GHz)')
    plt.ylabel('Flux density (Jy)')
    plt.grid(True)
    if (fitsHeader):
        titleText = 'Target: %s, OD: %i, OBSID: %i'%(fitsHeader['OBJECT'],fitsHeader['ODNUMBER'],fitsHeader['OBS_ID'])
        fig.suptitle(titleText, fontsize=14, fontweight='bold')
        #plt.title(titleText)
#
plotSpireSparseSpec(spss, fitsHeader=hh)
#
# add the off axis detectors to the plot
plotSpireSparseSpec(spss, fitsHeader=hh, onlyCentral=False)


What's next?

Use the provided defs and implement:

  • A background subtraction function.
  • Pointing offset corrector.
  • Semi-extended corrector.
  • FTS footprint plot.

Short-term implementations:

  • Provide similar dictionary for the mapping observations: preCubes and cubes.

Long-term ideas:

  • Create special class for FTS sparse spectra, even better, think about storing the spectral data as HDF5 object.
  • Special class for FTS pre-cubes and spectral cubes.
  • Read access to the non-averaged spectral products, i.e. for each detector and each spectral scan.

Ivan Valtchanov, Herschel Science Centre, March 2016


In [ ]: