Does a good framework for MongoDB "schema" upgrades in Scala exist?

Question

What are the options for MongoDB schema migrations/upgrades?

We (my colleagues and I) have a somewhat large (~100 million record) MongoDB collection. This collection is mapped (ORM'd) to a Scala lift-mongodb object that has been through a number of different iterations. We've got all sorts of code in there which handles missing fields, renames, removals, migrations, etc.

As much as the whole "schema-less" thing can be nice and flexible, in this case it's causing a lot of code clutter as our object continues to evolve. Continuing down this "flexible object" path is simply not sustainable.

How have you guys implemented schema migrations/upgrades in MongoDB with Scala? Does a framework for this exist? I know that Foursquare uses Scala with MongoDB and Rogue (their own query DSL)... does anyone know how they handle their migrations?

Thank you.

Telesthesia

What you are storing that has 100 million records.. purely curious... I love to see where/how technology is used – Localize

@AdamGent: global coverage of stock corporate action data (dividends and splits), going back for quite a while. Next up might be tick-history with 1-second-ish resolution... that's way more data, and a whole different can of worms. – Telesthesia

We do data migration with fairly straightforward (versioned) JavaScript scripts and a schema version document in a reserved collection to automatically trigger the appropriate migrations. – Mackmackay

<nitpick>technically, MongoDB<->object mapping is not Object Relational Mapping</nitpick> :P – Diplomatic

Answer 1

Perhaps this can help somewhat, this is how Guardian.co.uk handle this:

http://qconlondon.com/dl/qcon-london-2011/slides/MatthewWall_WhyIChoseMongoDBForGuardianCoUk.pdf

Schema upgrades

This can be mitigated by:

• Adding a “version” key to each document
• Updating the version each time the application modifies a document
• Using MapReduce capability to forcibly migrate documents from older versions if required

Superfuse

the external link is not valid any more, but thanks that you mentioned the main points here – Hartwell

Answer 2

I program migrations of MongoDB data with my own Scala framework "Subset". It lets define document fields pretty easily, fine tune data serialization (e.g. write "date" is a specific format and so on) and build queries and update modifiers in terms of the defined fields. This "gist" gives a good introduction

Challenging