How can I read hdf5 files stored as 1-D array. and view them as images?

Question

I have a large image classification dataset stored in the format .hdf5. The dataset has the labels and the images stored in the .hdf5 file. I am unable to view the images as they are store in form of an array. The dataset reading code that I have used is as follows,

import h5py
import numpy
f = h5py.File('data/images.hdf5', 'r')
print(list(f.keys()))

['datasets']

group = f['datasets']
list(group.keys())

['car']

Now when I read the group cars I have the following output,

data = group['car']
data.shape,data[0].shape,data[1].shape

((51,), (383275,), (257120,)

So it looks like there are 51 images for label car and images are stored as 383275 and 257120 dimensional arrays, with no information about their height and width dimensions. I want to save the images as RGB again. Next following the code here, I tried to read the images.

import numpy as np
from PIL import Image

# hdf = h5py.File("Sample.h5",'r')
array = data[0]
img = Image.fromarray(array.astype('uint8'), 'RGB')
img.save("yourimage.thumbnail", "JPEG")
img.show()

Unfortunately, the following error is received.

File /usr/local/lib/python3.8/dist-packages/PIL/Image.py:784, in Image.frombytes(self, data, decoder_name, *args)
    781 s = d.decode(data)
    783 if s[0] >= 0:
--> 784     raise ValueError("not enough image data")
    785 if s[1] != 0:
    786     raise ValueError("cannot decode image data")

ValueError: not enough image data

References I have already checked the hdf group help library etc. Any help will be highly useful. Thanks.

Schleswigholstein

Answer 1

First, the f['datasets']['car'] object is a DATASET, not a GROUP. Second, based this output, I think your dataset is a dataset with rows of variable length arrays (aka "ragged" array).

# this is group object reference:
group = f['datasets']
# these are equivalent dataset object references:
data = group['car']  
data = f['datasets']['car']  
# this gives the dataset shape (# of rows), then the shape for data on row 0 and  row 1:
data.shape,data[0].shape,data[1].shape
((51,), (383275,), (257120,)

From this, I think you have 51 rows, each with a single 1-D array of different sizes. If I'm correct, the original image array was flattened.

To reconstruct each image you will need the original shape of the image. Hopefully it's documented somewhere in the file. Do you have the schema definition for this file? If not, you will have to determine that somehow. [If so, someone didn't do you any favors. This is an example of something "clever", but not very helpful for future users.]

Hopefully they left you some "bread crumbs" to get each image size. As @Manoj Bhosle suggests, there might be in another dataset. Or, they could be saved as attributes. Either way, you will have to interrogate the file to figure this out. The easiest way is to open it with HDFView and examine it. HDFView can be downloaded from The HDF Group.

You can access any attributes on the 'car' dataset with this code:

with h5py.File('data/images.hdf5') as h5f:
    ds = h5f['datasets']['car']
    for k in ds.attrs.keys():
        print(f"{k} => {ds.attrs[k]}")

You can check the size of the array stored on each row in the 'car' dataset with this code:

with h5py.File('data/images.hdf5') as h5f:
    ds = h5f['datasets']['car']
    for row in ds:
        print(row.shape, row[0].shape)

Portable

Thanks a lot. Yes it looks like the attributes are missing. – Schleswigholstein

That's unfortunate. I wonder how anyone can recover the images without them?? (rhetorical question) I had another idea -- calculate the factors of the flatten array size, then reshape it with the "most likely" values. I calculated factors for your 2 values (257_120 & 383_275), but didn't get any factors that looked promising. – Portable

Answer 2

import h5py
import numpy as np
from PIL import Image

# Open the HDF5 file
with h5py.File('data/images.hdf5', 'r') as f:
    # Access the dataset containing images
    data = f['datasets']['car']

    # Assuming there's a corresponding dataset for dimensions
    dimensions = f['datasets']['dimensions']

    # Iterate through each image stored as a 1-D array
    for i in range(len(data)):
        # Retrieve the dimensions for the current image
        height, width, channels = dimensions[i]

        # Reshape the 1-D array into a 3-D array with the correct shape
        image_array = np.reshape(data[i], (height, width, channels))

        # Convert the numpy array into a PIL image
        img = Image.fromarray(image_array.astype('uint8'), 'RGB')

        # Save or display the image
        img.save(f'image_{i}.png')
        # Remove the img.show() line if you don't want to display the image
        img.show()

#may this helps to you

Siebert

Thanks a lot for your approach. The attributes seem to missing for the datasets. – Schleswigholstein

Answer 3

import h5py
import numpy as np
from PIL import Image

# Open the HDF5 file
with h5py.File('data/images.hdf5', 'r') as f:
    # Access the dataset containing images
    data = f['datasets']['car']

    # Iterate through each image stored as a 1-D array
    for i in range(len(data)):
        # Assuming you know the correct dimensions. For example, 256x256 with 3 color channels
        height, width, channels = 256, 256, 3

        # Reshape the 1-D array into a 3-D array with the shape (height, width, channels)
        image_array = np.reshape(data[i], (height, width, channels))

        # Convert the numpy array into a PIL image
        img = Image.fromarray(image_array.astype('uint8'), 'RGB')

        # Save or display the image
        img.save(f'image_{i}.png')
        img.show()  # Remove this line if you don't want to display the image

Siebert

Unfortunately the height, width, channels dimnensions of the resized image are not available. That is why I am unable to reshape my images. The sizes are also different for each image. – Schleswigholstein

That is why your solution will not work. – Schleswigholstein

@Schleswigholstein please check updated solution – Siebert