Sparse array support in HDF5

Question

I need to store a 512^3 array on disk in some way and I'm currently using HDF5. Since the array is sparse a lot of disk space gets wasted.

Does HDF5 provide any support for sparse array ?

Anschluss

[This answer][1] provides a (Python only) solution. [1]: https://mcmap.net/q/594207/-storing-numpy-sparse-matrix-in-hdf5-pytables – Bypath

Answer 1

Chunked datasets (H5D_CHUNKED) allow sparse storage but depending on your data, the overhead may be important.

Take a typical array and try both sparse and non-sparse and then compare the file sizes, then you will see if it is really worth.

Flotsam

yes, this post explains how to do it (or perhaps how not to do it) thanks – Anschluss

Answer 2

One workaround is to create the dataset with a compression option. For example, in Python using h5py:

import h5py
f = h5py.File('my.h5', 'w')
d = f.create_dataset('a', dtype='f', shape=(512, 512, 512), fillvalue=-999.,
                     compression='gzip', compression_opts=9)
d[3, 4, 5] = 6
f.close()

The resulting file is 4.5 KB. Without compression, this same file would be about 512 MB. That's a compression of 99.999%, because most of the data are -999. (or whatever fillvalue you want).

---

The equivalent can be achieved using the C++ HDF5 API by setting H5::DSetCreatPropList::setDeflate to 9, with an example shown in h5group.cpp.

Anthropology

Although the analysis is indeed done in python, the hdf5 file is generated in C++ so h5py is not an option. Is the same kind of compression supported natively by hdf5? I know pytables and h5py support additional compression protocols. – Anschluss

@Anschluss I've updated the answer with a link to a C++ example that does the same technique. I do believe that the dataset must be chunked to enable compression. – Anthropology

From the HDF5 link at the start of the answer: "Chunked storage makes it possible to resize datasets, and because the data is stored in fixed-size chunks, to use compression filters." So, yep, chunking required for compression. – Gordy

What does the compression_opts do in the above code? – Antifouling

@Antifouling 9 is the maximum compression level; see the docs – Anthropology

Answer 3

HDF5 provides indexed storage: http://www.hdfgroup.org/HDF5/doc/TechNotes/RawDStorage.html

Porcelain

hi, Ím not really familiar with how hdf5 works internally, how can I store raw data in a hdf5 file ? does that mean I can bypass the Table datatype and write my own structures ? – Anschluss