Python Record IO

In image_io we already learned how to pack image into standard recordio format and load it with ImageRecordIter. This tutorial will walk through the python interface for reading and writing record io files. It can be useful when you need more more control over the details of data pipeline. For example, when you need to augument image and label together for detection and segmentation, or when you need a custom data iterator for triplet sampling and negative sampling.

Setup environment first:



In [26]:

    
%matplotlib inline
from __future__ import print_function
import mxnet as mx
import numpy as np
import matplotlib.pyplot as plt

The relevent code is under mx.recordio. There are two classes: MXRecordIO, which supports sequential read and write, and MXIndexedRecordIO, which supports random read and sequential write.

MXRecordIO

First let's take a look at MXRecordIO. We open a file tmp.rec and write 5 strings to it:



In [14]:

    
record = mx.recordio.MXRecordIO('tmp.rec', 'w')
for i in range(5):
    record.write('record_%d'%i)
record.close()

Then we can read it back by opening the same file with 'r':



In [15]:

    
record = mx.recordio.MXRecordIO('tmp.rec', 'r')
while True:
    item = record.read()
    if not item:
        break
    print(item)
record.close()









    



record_0
record_1
record_2
record_3
record_4

MXIndexedRecordIO

Some times you need random access for more complex tasks. MXIndexedRecordIO is designed for this. Here we create a indexed record tmp.rec and a corresponding index file tmp.idx:



In [16]:

    
record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w')
for i in range(5):
    record.write_idx(i, 'record_%d'%i)
record.close()

We can then access records with keys:



In [17]:

    
record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'r')
record.read_idx(3)









    Out[17]:





'record_3'

You can list all keys with:



In [18]:

    
record.keys









    Out[18]:





[0, 1, 2, 3, 4]

Packing and Unpacking Data

Each record in a .rec file can contain arbitrary binary data, but machine learning data typically has a label/data structure. mx.recordio also contains a few utility functions for packing such data, namely: pack, unpack, pack_img, and unpack_img.

Binary Data

pack and unpack are used for storing float (or 1d array of float) label and binary data:



In [27]:

    
# pack
data = 'data'
label1 = 1.0
header1 = mx.recordio.IRHeader(flag=0, label=label1, id=1, id2=0)
s1 = mx.recordio.pack(header1, data)
print('float label:', repr(s1))
label2 = [1.0, 2.0, 3.0]
header2 = mx.recordio.IRHeader(flag=0, label=label2, id=2, id2=0)
s2 = mx.recordio.pack(header2, data)
print('array label:', repr(s2))









    



float label '\x00\x00\x00\x00\x00\x00\x80?\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00data'
array label '\x03\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80?\x00\x00\x00@\x00\x00@@data'



In [30]:

    
# unpack
print(*mx.recordio.unpack(s1))
print(*mx.recordio.unpack(s2))









    



(HEADER(flag=0, label=1.0, id=1, id2=0), 'data')
(HEADER(flag=3, label=array([ 1.,  2.,  3.], dtype=float32), id=2, id2=0), 'data')

Image Data

pack_img and unpack_img are used for packing image data. Records packed by pack_img can be loaded by mx.io.ImageRecordIter.



In [36]:

    
# pack_img
data = np.ones((3,3,1), dtype=np.uint8)
label = 1.0
header = mx.recordio.IRHeader(flag=0, label=label, id=0, id2=0)
s = mx.recordio.pack_img(header, data, quality=100, img_fmt='.jpg')
print(repr(s))









    



'\x00\x00\x00\x00\x00\x00\x80?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00C\x00\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\xff\xc0\x00\x0b\x08\x00\x03\x00\x03\x01\x01\x11\x00\xff\xc4\x00\x1f\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\xff\xc4\x00\xb5\x10\x00\x02\x01\x03\x03\x02\x04\x03\x05\x05\x04\x04\x00\x00\x01}\x01\x02\x03\x00\x04\x11\x05\x12!1A\x06\x13Qa\x07"q\x142\x81\x91\xa1\x08#B\xb1\xc1\x15R\xd1\xf0$3br\x82\t\n\x16\x17\x18\x19\x1a%&\'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\x83\x84\x85\x86\x87\x88\x89\x8a\x92\x93\x94\x95\x96\x97\x98\x99\x9a\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xff\xda\x00\x08\x01\x01\x00\x00?\x00\xfe\x01\xeb\xff\xd9'



In [37]:

    
# unpack_img
print(*mx.recordio.unpack_img(s))









    



HEADER(flag=0, label=1.0, id=0, id2=0) [[1 1 1]
 [1 1 1]
 [1 1 1]]

Next Step

Advanced Image IO Advanced image IO for detection, segmentation, etc...



In [ ]: