Selecting between shelve and sqlite for really large dictionary (Python)

Question

I have a large Python dictionary of vectors (150k vectors, 10k dimensions each) of float numbers that can't be loaded into memory, so I have to use one of the two methods for storing this on disk and retrieving specific vectors when appropriate. The vectors will be created and stored once, but might be read many (thousands of) times -- so it is really important to have efficient reading. After some tests with shelve module, I tend to believe that sqlite will be a better option for this kind of task, but before I start writing code I would like to hear some more opinions on this... For example, are there any other options except of those two that I'm not aware of?

Now, assuming we agree that the best option is sqlite, another question relates to the exact form of the table. I'm thinking of using a fine-grained structure with rows of the form vector_key, element_no, value to help efficient pagination, instead of storing all 10k elements of a vector into the same record. I would really appreciate any suggestions on this issue.

Bathymetry

Answer 1

You want sqlite3, then if you use an ORM like sqlalchemy then you can easily grow to expand and use other back end databases.

Shelve is more of a "toy" than actually useful in production code.

The other point you are talking about is called normalization and I have personally never been very good at it this should explain it for you.

Just as an extra note this shows performance failures in shelve vs sqlite3

Pampas

Hi, I've seen before the link you're suggesting, the problem is that it tests only writing -- not reading which is my highest concern. As for normalization, both forms I suggest above are already in normal form, I just wonder which of the two will be the most efficient for reading -- I guess I can perform some tests on this. In any case thank you for the answer :) – Bathymetry

Answer 2

As you are dealing with numeric vectors, you may find PyTables an interesting alternative.

Dualpurpose

Hi, it seems indeed as an interesting alternative, I'll have a better look at it. Thank you for the suggestion :) – Bathymetry