Adding mongoDB document-array-element using Python Eve

Question

Background: (using Eve and Mongo)

I'm working in Python using the Eve REST provider library connecting and to a mongoDB to expose a number of REST endpoints from the database. I've had good luck using Eve so far, but I've run into a problem that might be a bit beyond what Eve can do natively.

My problem is that my mongoDb document format has a field (called "slots"), whose value is a list/array of dictionaries/embedded-documents.

So the mongoDB document structure is:

{
   blah1: data1,
   blah2: data2,
   ...
   slots: [
       {thing1:data1, thing2:data2},
       {thingX:dataX, thingY:dataY}
   ]
}

I need to add new records (I.E. add pre-populated dictionaries) to the 'slots' list.

If I imagine doing the insert directly via pymongo it would look like:

mongo.connection = MongoClient()
mongo.db = mongo.connection['myDB']
mongo.coll = mongo.db['myCollection']

...

mongo.coll.update({'_id' : document_id}, 
                  {'$push': { "slot" : {"thing1":"data1","thingX":"dataX"}  }  } )

The REST action/URI combo that I would like to do this action is a POST to '_id/slots', e.g. URI of /app/012345678901234567890123/slots.

Problem: (inserting an element into an array in Eve)

From SO: How to add to a list type in Python Eve without replacing old values and eve project issue it appears Eve doesn't currently support operating on mongoDB embedded documents (or arrays?) unless the entire embedded document is rewritten, and rewriting the whole array is very undesirable in my case.

---

So, assuming its true Eve doesn't have a method to allow inserting of array elements (and given I already have numerous other endpoints working well inside of Eve)...

---

... I'm now looking for a way, inside of an Eve/Flask configuration with multiple working endpoints, to intercept and change Eve's mongoDB write for just this one endpoint.

I know (worst case) I can override the routing of Eve and to completely do the write by hand, but then I would have manage the _updated and hand check & change the documents _etag value, both things I would certainly prefer not to have to write new code for.

I've looked at Eve's Datebase event hooks but I don't see a way to modify the database commands that are executed (I can see how to change the data, but not the commands).

Anyone else already solved this problem already? If not any ideas on the most direct path to implement by hand? (hopefully reusing as much of Eve as possible because I do want to continue using Eve for all my (already working) endpoints)

Flaggy

Answer 1

This is an interesting question. I believe that in order to achieve your goal you would need to perform two actions:

1. Build and pass a custom Validator.
2. Build and pass a custom Mongo data layer.

This might sound like too much work, but that's probably not the case.

---

Custom Validator

A custom validator is going to be needed because when you perform your PATCH request on the "push-enabled" endpoint you want to pass a document which is syntactically different from endpoint validation schema. You are going to pass a dict ({"slot": {"thing1": "data1", "thingX": "dataX"}}) whereas the endpoint expects a list:

'mycollection': {
    'type': 'list',
    'schema': {
        'type': 'dict',
        'schema': {
            'thing1': {'type': 'string'},
            'thingX': {'type': 'string'},
        }
    }
}

If you don't customize validation you will end up with a validation error (list type expected). I guess your custom validator could look something like:

from eve.data.mongo.validation import Validator
from flask import request

class MyValidator(Validator):
    def validate_replace(self, document, _id, original_document=None):
        if self.resource = 'mycollection' and request.method = 'PATCH':
            # you want to perform some real validation here
            return True
        return super(Validator, self).validate(document)

Mind you I did not try this code so it might need some adjustment here and there.

An alternative approach would be to set up an alternative endpoint just for PATCH requests. This endpoint would consume the same datasource and have a dict-like schema. This would avoid the need for a custom validator and also, you would still have normal atomic field updates ($set) on the standard endpoint. Actually I think I like this approach better, as you don't lose functionality and reduce complexity. For guidance on multiple endpoints hitting the same datasource see the docs

---

Custom data layer

This is needed because you want to perform a $push instead of a $set when mycollection is involved in a PATCH request. Something like this maybe:

from eve.io.mongo import Mongo
from flask import request

class MyMongo(Mongo):
    def update(self, resource, id_, updates, original):
        op = '$push' if resource == 'mycollection' else '$set'
        return self._change_request(resource, id_, {op: updates}, original)

---

Putting it all together

You then use your custom validator and data layers upon app initialisation:

app = Eve(validator=MyValidator, data=MyMongo)
app.run()

Again I did not test all of this; it's Sunday and I'm on the beach so it might need some work but it should work.

With all this being said, I am actually going to experiment with adding support for push updates to the standard Mongo data layer. A new pair of global/endpoint settings, like MONGO_UPDATE_OPERATOR/mongo_update_operator are implemented on a private branch. The former defaults to $set so all API endpoints still perform atomic field updates. One could decide that a certain endpoint should perform something else, say a $push. Implementing validation in a clean and elegant way is a little tricky but, assuming I find the time to work on it, it is not unlikely that this could make it to Eve 0.6 or beyond.

Hope this helps.

Underfur

Thank you for the excellent answer! I'm reading through and hope to implement in the next few days. One question: why did you transform my "POST" into a "PATCH". From a REST point of view, I'm adding a new record (albeit in the DB it's a new sub record but the user of the REST API doesn't know that), so a POST (add new) seems more logical then a PATCH (modify existing), don't you agree? Would your answer still apply if the POST logic is used instead? Thanks again! – Flaggy

Well you use $push is an update operator in mongo, you can't use it with an insert, so performing POST (insert) with a push would not be possible. – Underfur

Unfortunately this answer doesn't work as written. The updates dictionary passed to def update has both the values from the REST PATCH and the updates to _etag and _updated fields, with the former needing a $push and the latter needing to be $set. I'm attempting a work-around which I'll post if i get it to work, but its going to be ugly. – Flaggy

In your updated you could split the payload and then invoke _change_request twice, once for the push part and another for the set. Not ideal from a performance standpoint but and keep in mind that etag is going to be an issue anyway since the pre-computed one is not going to be representative of the final stored document (because of your changes). You might need to re-compute it. – Underfur

@NicolaIarocci Did you ever add support for 'push updates' to Eve? – Embroidery

@MikeLutz It's been 3 years but did you ever figure this out? – Embroidery

@Embroidery I've searched through my eve codebases and I don't see any solutions implemented as above - been a long time since I've thought about this code, so I can't quite remember what I'm reading. Apologies for not updating as I said I would – Flaggy

@MikeLutz Mike - thanks for checking back in. I ended up writing my own flask-app route and talking through pymongo directly. Sadly the 'append to array' performance degraded over time so we skipped that technique. – Embroidery

Answer 2

I solved this a different way. I created a second domain that had each individual item in the array. Then, I hooked up to the insert events of Eve http://python-eve.org/features.html#insert-events. In the post insert event handler, I then grabbed the item and used code like this:

    return current_app.data.driver.db['friends'].update(
            { 'id': items[0]['_friend_id']  },
            {
                 '$push': { 'enemies': {'enemy':items[0]['enemy'],

                 }
                 },
                    '$currentDate': { 'lastModified': True }
            }
    )

Basically I now have normalized and denormalized views in the db.

Froward