Parsing Yann LeCun's MNIST IDX file format

Question

I would like to understand how to open this version of the MNIST data set. For example, the training set label file train-labels-idx1-ubyte is defined as:

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

And I found some code online that seems to work, but do not understand how it works:

with open('train-labels-idx1-ubyte', 'rb') as f:
    bytes = f.read(8)
    magic, size = struct.unpack(">II", bytes)

print(magic) # 2049
print(size)  # 60000

My understanding is that struct.unpack interprets the second argument as a big-endian byte string of two 4-byte integers (See here). When I actually print the value of bytes, though, I get:

b'\x00\x00\x08\x01\x00\x00\xea`'

The first four-byte integer makes sense:

b'\x00\x00\x08\x01'

The first two bytes are 0. The next indicates the data are unsigned bytes. And 0x01 indicates a 1-dimensional vector of labels. Assuming my understanding is correct so far, what is happening with the next three (four?) bytes:

...\x00\x00\xea`

How does this translate to 60,000?

Slicker

Can I suggest that you change the title of the question? IMO it is misleading. Yhe question is not how to parse the MNIST dataset, but to understand how a bytes sequence from a file becomes an integer number. – Avebury

You can find an already decoded version of the MNIST dataset here: mnist-decoded.000webhostapp.com – Raiment

Answer 1

I wrote the following code in case anyone needs to parse the whole dataset of images (as it appears in the question's title), and not just the first two bytes.

import numpy as np
import struct

with open('samples/t10k-images-idx3-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    nrows, ncols = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size, nrows, ncols))

This assumes you uncompressed the .gz file. You can also work with the compressed file, as indicated by Marktodisco's answer, by adding import gzip, using gzip.open(...) instead of open(...), and using np.frombuffer(f.read(), ...) instead of np.fromfile(f, ...).

And just to check, show the first digit. In my case it's a 7.

import matplotlib.pyplot as plt
plt.imshow(data[0,:,:], cmap='gray')
plt.show()

In addition, the following code reads the file with labels

with open('samples/t10k-labels-idx1-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size,)) # (Optional)
print(data)
# Prints: [7 2 1 ... 4 5 6]

The last reshape can be (size,) or (1, size) depending on your standards.

Fiddlededee

I've spent most of today trying to figure out how to convert from CSV to idx binary/MNIST; would you mind having a look at my question? #65143825 (why I had to convert it to CSV at all is also addressed here: #65143470), if you could help with either I'd appreciate it, I feel like your answer above is what I want to do sort of, but in reverse. – Acanthaceous

Answer 2

To understand how it works, you need to convert it to its binary representation.

As you mentioned, Python is correctly extracting the right information:

>>> import struct
>>> with open('train-labels-idx1-ubyte', 'rb') as f:
...     data = f.read(8)
... 
>>> print(data)
b'\x00\x00\x08\x01\x00\x00\xea`'
>>> print(struct.unpack('>II', data))
(2049, 60000)

In the header of the string, there are two 4-bytes integers. We can see their binary and decimal representation if we iterate over data:

>>> for char in data:
...     print('{0:08b} - {0:3d} - {1:s}'.format(char, str(bytes([char]))))
... 
00000000 -   0 - b'\x00'
00000000 -   0 - b'\x00'
00001000 -   8 - b'\x08'
00000001 -   1 - b'\x01'
00000000 -   0 - b'\x00'
00000000 -   0 - b'\x00'
11101010 - 234 - b'\xea'
01100000 -  96 - b'`'

The easy part is to know that the first 4 bytes are the first integer (the magic number), and the next 4 bytes are the second integer (the number of items).

Then, given these last 4 bytes, there are two ways one can construct the integer value they represent.

The first option (the one used in MNIST), is big or high endian. Which means, that the MOST significant bytes are found first:

00000000 00000000 11101010 01100000

If you check the decimal value of this binary number, it is 60,000, the number of items in the MNIST dataset.

Also, we could interpret this as little endian. In this case, the LESS significant bytes are found first:

01100000 11101010 00000000 00000000

Which in its decimal representation, is the number 1,625,948,160.

So, if you simply convert each byte in \x00\x00\xea` to binary, and you find the decimal representation of that whole binary number (reverting the order of the bytes if little endian), you have the integer value they represent.

Avebury

Answer 3

Carlos's answer is great, but it breaks if the files are still in .gz format. When I run the code I get the following error:

ValueError: cannot reshape array of size 1648861 into shape (10000,28,28)

Since the raw data downloads with a .gz extension by default, I've modified Carlos's code. See below.

import gzip
import struct
import numpy as np

with gzip.open('t10k-images-idx3-ubyte.gz','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    nrows, ncols = struct.unpack(">II", f.read(8))
    data = np.frombuffer(f.read(), dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size, nrows, ncols))

And the images still load correctly.

import matplotlib.pyplot as plt

plt.imshow(data[0,:,:], cmap='gray')
plt.show()

Cuttlebone

Answer 4

Merging, this work for me:

def load_dataset(path_dataset):       
   with gzip.open(path_dataset,'rb') as f:
     magic, size = struct.unpack(">II", f.read(8))
     nrows, ncols = struct.unpack(">II", f.read(8))
     data = np.frombuffer(f.read(), dtype=np.dtype(np.uint8).newbyteorder('>'))
     data = data.reshape((size, nrows, ncols))
     return data

def load_label(path_label):
  with gzip.open(path_label,'rb') as f:
    magic, size = struct.unpack('>II', f.read(8))
    label = np.frombuffer(f.read(), dtype=np.dtype(np.uint8).newbyteorder('>'))
    return label
    
X = load_dataset(r'samples/train-images-idx3-ubyte.gz')
y = load_label(r'samples/train-labels-idx1-ubyte.gz')

Skvorak