Is there anything wrong with creating a Python Pickle powered website?

Question

I have been toying with this idea for quite awhile now, but haven't seen any information on people doing it. I have a small website project where I need to load and modify 1 object. This object is pretty simple, and shouldn't be more than a few kb. Instead of running a DB for this small amount of data, why not just use pickle and/or shelve to save this data, and load it? I am planning on using a micro web framework like Bottle or Flask for the project.

Are there any reasons to not use this method to load the data? It will only load the pickle file at the time Apache starts up, so I don't think speed will be effected (faster than querying a db).

Thanks for any input!

Internment

Why use pickle? Why not simple plain text? Why not Python source code for the object? Why limit yourself to pickling the object? – Luxuriate

Is there any reason not to use something like SQLite? – Delaine

@Luxuriate The object contains sub-objects, with some datetime objects in there. I could do plaintext if necessary. There is a page on the site to insert a bit of new data, so that's why it can't be in the source itself. I'm trying to keep the site simple, no DB if i don't need one. – Internment

Please update the question with additional facts. Please make the question as complete and correct as possible. – Luxuriate

Answer 1

There is no reason why you can't implement object persistence via the standard Python pickle or shelve modules. Just make sure your objects are cleanly and securely picklable. Scalability may become a concern if your site grows beyond your current scope, but until then your idea should work just fine. If that day comes, the next obvious step would be to consider using Python's excellent SQLite module that comes pre-packaged with recent versions of the language.

Cassella

Something like MongoDB would be an easier transition from Pickle than would SQL be. – Liggitt

Yep, there's nothing wrong with getting it up and running with pickle. And who knows, maybe pickle will suffice for your problem. If not, you can always switch over to a database when it becomes a problem. Keep it simple until it demands more. – Zayin

@David M I have worked with SQL a lot before so either wouldn't be a problem, I would just choose to use MongoDB to get more experience in NoSQL :) – Internment

I'm not saying that SQL is hard, just that MongoDB et al are more similar to pickle. – Liggitt

Re SQL vs. NoSQL: there is no generic right answer to this debate. It all devolves into personal preference and religion for all but the most extreme use-cases. Best to give both a try and benefit from the broader perspectives gained only from experience. – Cassella

Answer 2

I wouldn't write a pickled string to a file directly. There are too many low-level details to worry about. Check out Durus, ZODB, or this post from FriendFeed about storing Python objects in MySQL.

Don't discard relational databases, though, they give you a lot of bang right out of the box (even for simple projects).

Cralg

Thanks for the links. If I'm going to use any DB, it's going to be MongoDB. I did not think about the problem of the framework running multiple python processes and not having access to the same object if something is modified. It would have to trigger some kind of flag and have each process reload the pickled file? What are some other "low-level details to worry about"? Thanks! – Internment

Answer 3

In addition to the concurrency issues you are already aware of, you also must ensure that the file is always in a consistent state. For example, if the server crashes in the middle of writing the file, what happens then? It's a case you need to consider and implement a solution for if you go this route.

Ferricyanide