Instructions

  • pip install planetarypy

I wrote highly generalized PDS index readers, that wrap the Python Parameter Value Language module PVL (The syntax for Planetary Data System label files). Relevant functions are in https://github.com/michaelaye/planetarypy/blob/master/planetarypy/pdstools.py

But possibly it's more efficient that you just quickly parse the specific data files yourself:

Download cumulative RDR index and label file here:

The label file has the column names for the .TAB file, but in PVL format. The .TAB file is a fixed format text file. You can read out the column names and the column specification (i.e. start and end byte for each column) below, in case you can feed them to a Ruby text parser.


In [6]:
from planetarypy import pdstools as pds

replace local file names here:


In [7]:
labelfname = '/Users/klay6683/Dropbox/data/hirise/index/RDRCUMINDEX.LBL'
tablefname = '/Users/klay6683/Dropbox/data/hirise/index/RDRCUMINDEX.TAB'

In [8]:
label = pds.IndexLabel(labelfname)

In [9]:
label.colnames


Out[9]:
['VOLUME_ID',
 'FILE_NAME_SPECIFICATION',
 'INSTRUMENT_HOST_ID',
 'INSTRUMENT_ID',
 'OBSERVATION_ID',
 'PRODUCT_ID',
 'PRODUCT_VERSION_ID',
 'TARGET_NAME',
 'ORBIT_NUMBER',
 'MISSION_PHASE_NAME',
 'RATIONALE_DESC',
 'OBSERVATION_START_TIME',
 'OBSERVATION_START_COUNT',
 'START_TIME',
 'SPACECRAFT_CLOCK_START_COUNT',
 'STOP_TIME',
 'SPACECRAFT_CLOCK_STOP_COUNT',
 'IMAGE_LINES',
 'LINE_SAMPLES',
 'EMISSION_ANGLE',
 'INCIDENCE_ANGLE',
 'PHASE_ANGLE',
 'SPACECRAFT_ALTITUDE',
 'TARGET_CENTER_DISTANCE',
 'SLANT_DISTANCE',
 'NORTH_AZIMUTH',
 'SUB_SOLAR_AZIMUTH',
 'SUB_SOLAR_LATITUDE',
 'SUB_SOLAR_LONGITUDE',
 'SUB_SPACECRAFT_LATITUDE',
 'SUB_SPACECRAFT_LONGITUDE',
 'SOLAR_DISTANCE',
 'SOLAR_LONGITUDE',
 'LOCAL_TIME',
 'STEREO_FLAG',
 'MINIMUM_LATITUDE',
 'MAXIMUM_LATITUDE',
 'MINIMUM_LONGITUDE',
 'MAXIMUM_LONGITUDE',
 'MAP_SCALE',
 'MAP_RESOLUTION',
 'MAP_PROJECTION_TYPE',
 'PROJECTION_CENTER_LATITUDE',
 'PROJECTION_CENTER_LONGITUDE',
 'LINE_PROJECTION_OFFSET',
 'SAMPLE_PROJECTION_OFFSET',
 'CORNER1_LATITUDE',
 'CORNER1_LONGITUDE',
 'CORNER2_LATITUDE',
 'CORNER2_LONGITUDE',
 'CORNER3_LATITUDE',
 'CORNER3_LONGITUDE',
 'CORNER4_LATITUDE',
 'CORNER4_LONGITUDE']

In [10]:
label.colspecs


Out[10]:
[(1, 11),
 (14, 81),
 (84, 87),
 (90, 96),
 (99, 114),
 (117, 138),
 (141, 144),
 (147, 179),
 (181, 187),
 (189, 219),
 (222, 297),
 (300, 324),
 (327, 343),
 (346, 370),
 (373, 389),
 (392, 416),
 (419, 435),
 (437, 443),
 (444, 450),
 (451, 459),
 (460, 467),
 (468, 476),
 (477, 485),
 (486, 493),
 (494, 502),
 (503, 513),
 (514, 524),
 (525, 535),
 (536, 546),
 (547, 557),
 (558, 568),
 (569, 579),
 (580, 590),
 (591, 601),
 (603, 606),
 (608, 618),
 (619, 629),
 (630, 640),
 (641, 651),
 (652, 657),
 (658, 668),
 (670, 689),
 (691, 696),
 (697, 705),
 (706, 718),
 (719, 731),
 (732, 742),
 (743, 753),
 (754, 764),
 (765, 775),
 (776, 786),
 (787, 797),
 (798, 808),
 (809, 819)]

In [13]:
label.index_path


Out[13]:
PosixPath('/Users/klay6683/Dropbox/data/hirise/index/RDRCUMINDEX.TAB')

In [15]:
df = label.read_index_data()

In [16]:
df.info()


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 92758 entries, 0 to 92757
Data columns (total 54 columns):
VOLUME_ID                       92758 non-null object
FILE_NAME_SPECIFICATION         92758 non-null object
INSTRUMENT_HOST_ID              92758 non-null object
INSTRUMENT_ID                   92758 non-null object
OBSERVATION_ID                  92758 non-null object
PRODUCT_ID                      92758 non-null object
PRODUCT_VERSION_ID              92758 non-null int64
TARGET_NAME                     92758 non-null object
ORBIT_NUMBER                    92758 non-null int64
MISSION_PHASE_NAME              92758 non-null object
RATIONALE_DESC                  92758 non-null object
OBSERVATION_START_TIME          92758 non-null object
OBSERVATION_START_COUNT         92758 non-null object
START_TIME                      92758 non-null object
SPACECRAFT_CLOCK_START_COUNT    92758 non-null object
STOP_TIME                       92758 non-null object
SPACECRAFT_CLOCK_STOP_COUNT     92758 non-null object
IMAGE_LINES                     92758 non-null int64
LINE_SAMPLES                    92758 non-null int64
EMISSION_ANGLE                  92758 non-null float64
INCIDENCE_ANGLE                 92758 non-null float64
PHASE_ANGLE                     92758 non-null float64
SPACECRAFT_ALTITUDE             92758 non-null float64
TARGET_CENTER_DISTANCE          92758 non-null float64
SLANT_DISTANCE                  92758 non-null float64
NORTH_AZIMUTH                   92758 non-null float64
SUB_SOLAR_AZIMUTH               92758 non-null float64
SUB_SOLAR_LATITUDE              92758 non-null float64
SUB_SOLAR_LONGITUDE             92758 non-null float64
SUB_SPACECRAFT_LATITUDE         92758 non-null float64
SUB_SPACECRAFT_LONGITUDE        92758 non-null float64
SOLAR_DISTANCE                  92758 non-null float64
SOLAR_LONGITUDE                 92758 non-null float64
LOCAL_TIME                      92758 non-null float64
STEREO_FLAG                     92692 non-null object
MINIMUM_LATITUDE                92758 non-null float64
MAXIMUM_LATITUDE                92758 non-null float64
MINIMUM_LONGITUDE               92758 non-null float64
MAXIMUM_LONGITUDE               92758 non-null float64
MAP_SCALE                       92758 non-null float64
MAP_RESOLUTION                  92758 non-null float64
MAP_PROJECTION_TYPE             92758 non-null object
PROJECTION_CENTER_LATITUDE      92758 non-null float64
PROJECTION_CENTER_LONGITUDE     92758 non-null float64
LINE_PROJECTION_OFFSET          92758 non-null float64
SAMPLE_PROJECTION_OFFSET        92758 non-null float64
CORNER1_LATITUDE                92758 non-null float64
CORNER1_LONGITUDE               92758 non-null float64
CORNER2_LATITUDE                92758 non-null float64
CORNER2_LONGITUDE               92758 non-null float64
CORNER3_LATITUDE                92758 non-null float64
CORNER3_LONGITUDE               92758 non-null float64
CORNER4_LATITUDE                92758 non-null float64
CORNER4_LONGITUDE               92758 non-null float64
dtypes: float64(33), int64(4), object(17)
memory usage: 38.2+ MB

I have never seen the data-item "SOUTH_AZIMUTH". Possibly, this was confused with the item called "SUB-SOLAR AZIMUTH"? I put that column in there for now.

example on how to filter for HiRISE obsids (i.e. Planet4 image_names)


In [17]:
list_of_image_names = [
    'ESP_040246_0935',
    'ESP_039969_0935',
    'ESP_039824_0935',
    'ESP_039547_0935',
    'ESP_039468_0935',
    'ESP_038822_0935',
    'ESP_038625_0930',
    'ESP_038492_0935',
    'ESP_038215_0935',
    'ESP_038149_0935',
    'ESP_038110_0930',
    'ESP_037964_0935',
    'ESP_040311_0940',
    'ESP_040193_0940',
    'ESP_037977_0940',
    'ESP_037976_0940',
]

In [18]:
df.set_index('OBSERVATION_ID', inplace=True)

In [25]:
data = df.loc[list_of_image_names]
# second line for the fact that lots/all metadata exists for _RED and _COLOR channels
data = data[data.PRODUCT_ID.str.endswith('_COLOR')]

In [26]:
data.info()


<class 'pandas.core.frame.DataFrame'>
Index: 16 entries, ESP_040246_0935 to ESP_037976_0940
Data columns (total 53 columns):
VOLUME_ID                       16 non-null object
FILE_NAME_SPECIFICATION         16 non-null object
INSTRUMENT_HOST_ID              16 non-null object
INSTRUMENT_ID                   16 non-null object
PRODUCT_ID                      16 non-null object
PRODUCT_VERSION_ID              16 non-null int64
TARGET_NAME                     16 non-null object
ORBIT_NUMBER                    16 non-null int64
MISSION_PHASE_NAME              16 non-null object
RATIONALE_DESC                  16 non-null object
OBSERVATION_START_TIME          16 non-null object
OBSERVATION_START_COUNT         16 non-null object
START_TIME                      16 non-null object
SPACECRAFT_CLOCK_START_COUNT    16 non-null object
STOP_TIME                       16 non-null object
SPACECRAFT_CLOCK_STOP_COUNT     16 non-null object
IMAGE_LINES                     16 non-null int64
LINE_SAMPLES                    16 non-null int64
EMISSION_ANGLE                  16 non-null float64
INCIDENCE_ANGLE                 16 non-null float64
PHASE_ANGLE                     16 non-null float64
SPACECRAFT_ALTITUDE             16 non-null float64
TARGET_CENTER_DISTANCE          16 non-null float64
SLANT_DISTANCE                  16 non-null float64
NORTH_AZIMUTH                   16 non-null float64
SUB_SOLAR_AZIMUTH               16 non-null float64
SUB_SOLAR_LATITUDE              16 non-null float64
SUB_SOLAR_LONGITUDE             16 non-null float64
SUB_SPACECRAFT_LATITUDE         16 non-null float64
SUB_SPACECRAFT_LONGITUDE        16 non-null float64
SOLAR_DISTANCE                  16 non-null float64
SOLAR_LONGITUDE                 16 non-null float64
LOCAL_TIME                      16 non-null float64
STEREO_FLAG                     16 non-null object
MINIMUM_LATITUDE                16 non-null float64
MAXIMUM_LATITUDE                16 non-null float64
MINIMUM_LONGITUDE               16 non-null float64
MAXIMUM_LONGITUDE               16 non-null float64
MAP_SCALE                       16 non-null float64
MAP_RESOLUTION                  16 non-null float64
MAP_PROJECTION_TYPE             16 non-null object
PROJECTION_CENTER_LATITUDE      16 non-null float64
PROJECTION_CENTER_LONGITUDE     16 non-null float64
LINE_PROJECTION_OFFSET          16 non-null float64
SAMPLE_PROJECTION_OFFSET        16 non-null float64
CORNER1_LATITUDE                16 non-null float64
CORNER1_LONGITUDE               16 non-null float64
CORNER2_LATITUDE                16 non-null float64
CORNER2_LONGITUDE               16 non-null float64
CORNER3_LATITUDE                16 non-null float64
CORNER3_LONGITUDE               16 non-null float64
CORNER4_LATITUDE                16 non-null float64
CORNER4_LONGITUDE               16 non-null float64
dtypes: float64(33), int64(4), object(16)
memory usage: 6.8+ KB

In [29]:
# don't really have a center lat/lon in this index, so calculating a rough avg value
data['MEAN_LATITUDE'] = (data.MAXIMUM_LATITUDE + data.MINIMUM_LATITUDE) / 2
data['MEAN_LONGITUDE'] = (data.MAXIMUM_LONGITUDE + data.MINIMUM_LONGITUDE) / 2

In [30]:
translator = {
    'acquisition_date': 'START_TIME',
    'local_mars_time': 'LOCAL_TIME',
    'latitude': 'MEAN_LATITUDE',
    'longitude': 'MEAN_LONGITUDE',
    'range_to_target': 'TARGET_CENTER_DISTANCE',
    'original_image_scale': 'MAP_SCALE',
    'emission_angle': 'EMISSION_ANGLE',
    'phase_angle': 'PHASE_ANGLE',
    'solar_incidence': 'INCIDENCE_ANGLE',
    'solar_longitude': 'SOLAR_LONGITUDE',
    'north_azimuth': 'NORTH_AZIMUTH',
    'south_azimuth': 'SUB_SOLAR_AZIMUTH',
}

In [33]:
output = data[list(translator.values())]
output


Out[33]:
START_TIME SOLAR_LONGITUDE LOCAL_TIME NORTH_AZIMUTH SUB_SOLAR_AZIMUTH MAP_SCALE EMISSION_ANGLE PHASE_ANGLE MEAN_LONGITUDE INCIDENCE_ANGLE TARGET_CENTER_DISTANCE MEAN_LATITUDE
OBSERVATION_ID
ESP_040246_0935 2015-02-26T11:01:29 298.441 17.6489 9.1137 282.9040 0.50 0.04306 67.7613 99.15210 67.7348 3626.77 -86.40150
ESP_039969_0935 2015-02-04T20:58:45 285.305 17.6507 9.1491 282.7429 0.50 2.02578 66.8265 99.19095 65.4817 3628.17 -86.40410
ESP_039824_0935 2015-01-24T13:48:37 278.303 17.8486 9.0735 279.6571 0.25 1.15364 65.7976 99.11390 64.9992 3626.63 -86.39990
ESP_039547_0935 2015-01-02T23:51:21 264.753 17.9796 8.8423 277.4356 0.25 2.23289 66.6109 98.88435 64.9539 3625.59 -86.38375
ESP_039468_0935 2014-12-27T20:07:24 260.859 18.2944 9.1944 273.1121 0.25 0.06100 65.4348 99.23225 65.4768 3626.56 -86.40530
ESP_038822_0935 2014-11-07T12:10:13 229.003 19.1478 9.1086 260.7446 0.50 2.71885 70.1466 99.13945 72.3592 3626.24 -86.39940
ESP_038625_0930 2014-10-23T03:58:16 219.442 20.8947 9.0660 235.0700 0.25 4.09091 73.0044 99.06175 76.3833 3628.50 -86.99520
ESP_038492_0935 2014-10-12T19:23:20 213.066 19.0108 9.0648 263.0736 0.50 0.17701 77.3975 99.10085 77.5370 3626.26 -86.39690
ESP_038215_0935 2014-09-21T05:35:22 200.035 19.1252 9.0180 261.6432 0.50 0.79762 82.0161 99.05330 82.6754 3626.81 -86.39550
ESP_038149_0935 2014-09-16T02:11:08 196.985 19.0249 8.9289 263.1284 0.50 0.19024 83.9984 98.96605 83.8261 3626.82 -86.39075
ESP_038110_0930 2014-09-13T01:15:26 195.194 20.0513 8.9274 247.8821 0.50 0.40143 84.8043 98.94500 85.1385 3626.97 -86.99070
ESP_037964_0935 2014-09-01T16:16:16 188.562 18.3581 8.7205 273.0602 0.50 7.38698 92.9270 98.77280 86.7101 3626.70 -86.40895
ESP_040311_0940 2015-03-03T12:35:10 301.471 16.3853 16.0353 308.6702 0.50 9.21077 72.8324 106.08500 67.0334 3626.72 -85.77840
ESP_040193_0940 2015-02-22T07:54:37 295.955 17.6870 16.1568 289.0824 0.25 6.51663 63.2444 106.19100 67.2161 3626.87 -85.78255
ESP_037977_0940 2014-09-02T16:34:57 189.148 18.4831 15.6649 278.0979 0.50 1.12302 85.7877 105.70600 86.6758 3626.58 -85.67555
ESP_037976_0940 2014-09-02T14:43:05 189.103 16.6881 15.9271 305.2541 0.50 23.89540 104.6830 106.00700 84.6922 3627.85 -85.66825

In [34]:
output.to_csv('planet4_metadata.csv')

In [35]:
!open .

In [ ]: