How to persist data between executions in Python

Question

I am working on a personal project in Python where I need some form of persistent data. The data would fit in 2-3 tables of 10-20 columns and 100-200 records each. I have a basic understanding of SQL, so a database seems to make some sense.

I am new to Python, so I am not familiar with the options for database interface from Python. I have also heard about pickling and am not sure if that would be a better solution for my project size. Can anyone recommend a good solution?

Saladin

Answer 1

Or, if you just want to persist data between executions - for such a small data set you could have a look at the pickle module for persistency, and just load the data into memory during execution.

It's a simple solution - but for a personal project it might be enough.

Chlamys

+1 I'd also start with pickle, just throw a bunch of lists of tuples at it. – Maillot

I think I will start with pickle. Thanks to all for the suggestions and information about different solutions! – Saladin

Answer 2

You should use sqlite3 module for this, it is included in Python.

Also you may want too look for an ORM solution.

Talithatalk

Answer 3

This sounds like very few data. An SQL DB might be overkill, especially with an ORM on top. I'd check whether JSON could do the job...

Ringhals

what are the advantages of JSON over pickle? From my cursory understanding, it seems like it would be better to use Python's own pickle format. – Saladin

It's widely supported (i.e. practically platform/language-independent), plus it's human readable. – Ringhals

But Json is not persistent on its own. You need to write and read it to a file/files, serialize/deserialize on reads/writes. Pickling might be faster on that. But pickling really pays when you have more complex data types to persist, like class instances. – Maillot

@Maillot Agreed. In case of small, simple data I would however opt for json - I just like the flexibility that comes with txt files, plus I might be a bit biased anyway as I use JS at times in addition... – Ringhals

json is pretty much the exact same code as pickle, so I wouldn't necessarily choose one because it's "easier" or anything. Pickle can handle more types, but it's not as human readable as json and it's limited to python. You could also consider YAML, which is similar to json but even more human readable (IMO). – Supersensible

Answer 4

I agree with using sqlite3. It is very easy to use, you don't need to worry about having to set up a database server. You should check out the SQLAlchemy library too.

Flavory

Answer 5

The real question is really what kind of operations you want to do with your data.

As far as storage possibilities, the simplest solutions are indeed sqlite3 and pickle.

The solution that you will choose depends basically on whether using SQL or Python is the easiest way for you to manage your data. SQL is probably better at complex operations than Python, but Python is definitely more lightweight and simpler, and therefore is a good choice for simple operations. So, if using pickle+Python is too cumbersome, then sqlite3 is a very good choice.

Yonkers

Answer 6

Peewee is another ORM that works with SQLite. It is an alternative to SQLAlchemy. If using SQLite, I would consider Peewee for pet projects and SQLAlchemy for professional work. I typically would not use SQLite directly.

Drynurse