MNIST

Functions to load MNIST data.



In [1]:

    
using FixedPointNumbers;

File formats

The following is taken from THE MNIST DATABASE of handwritten digits:

The data is stored in a very simple file format designed for storing vectors and multidimensional matrices. General info on this format is given at the end of this page, but you don't need to read that to use the data files.

All the integers in the files are stored in the MSB first (high endian) format used by most non-Intel processors. Users of Intel processors and other low-endian machines must flip the bytes of the header.

There are 4 files:

train-images-idx3-ubyte: training set images
train-labels-idx1-ubyte: training set labels
t10k-images-idx3-ubyte: test set images
t10k-labels-idx1-ubyte: test set labels

The training set contains 60000 examples, and the test set 10000 examples.

The first 5000 examples of the test set are taken from the original NIST training set. The last 5000 are taken from the original NIST test set. The first 5000 are cleaner and easier than the last 5000.

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
…
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  60000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
…
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
…
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  10000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
…
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

Magic numbers



In [2]:

    
const MNIST_LABELS_MAGIC = 0x00000801;
const MNIST_IMAGES_MAGIC = 0x00000803;

Read functions

MNIST_read_labels

This function returns a vector of Int8, in the range 0 to 9.



In [3]:

    
function MNIST_read_labels(filename :: AbstractString)
    open(filename) do io
        magic = ntoh(read(io, UInt32))

        if magic != MNIST_LABELS_MAGIC
            error("File \"$filename\" contains the wrong magic number for an MNIST labels file, $magic")
        end

        n = ntoh(read(io, UInt32))

        return read(io, Int8, n)
    end
end;

MNIST_read_images

This function returns an array of UFixed8, where each row is the pixels for a given image, arranged row-wise.

The pixel coding is also inverted from the MNIST file, as in Julia, 0 is interpreted as black, not white.

Finally, an intercept column of ones is prepended to the images.



In [4]:

    
function MNIST_read_images(filename :: AbstractString)
    open(filename) do io
        magic = ntoh(read(io, UInt32))

        if magic != MNIST_IMAGES_MAGIC
            error("File \"$filename\" contains the wrong magic number for an MNIST images file, $magic")
        end

        n = ntoh(read(io, UInt32))

        rows = Int(ntoh(read(io, UInt32)))
        cols = Int(ntoh(read(io, UInt32)))

        images = Array{UFixed8}(n, 1 + rows * cols)

        for i in 1:n
            images[i, 1] = 1
            images[i, 2:end] = UFixed8(1) - vec(read(io, UFixed8, (cols, rows)))
        end

        return (cols, rows, images)
    end
end;