Update a MongoEngine document using a python dict?

Question

Is it possible to update a MongoEngine document using a python dict?

For example:

class Pets(EmbeddedDocument):
    name = StringField()

class Person(Document):
    name = StringField()
    address = StringField()
    pets = ListField(EmbeddedDocumentField(Pets))

p = Person()
p.update_with_dict({
    "name": "Hank",
    "address": "Far away",
    "pets": [
        {
            "name": "Scooter"
        }
    ]
})

Prop

Could you be more clear with your problem please. From reading you answer below it became clear you are looking for a solution that allows you to update EmbeddedDocuments though this should be be stated in the question. – Vision

@Vision No, I want to update the complete document including embedded documents/listfields etc using a Python dict. – Prop

Instead of update, do you mean create? p = Person() instantiates the object with default and doesn't fetch an existing object from the DB to then update the attributes. – Dowlen

Answer 1

Ok I just made a function for it.

You call it like update_document(document, data_dict). It will loop through the items of data_dict and get the field instance using the key of the data_dict. It will then call field_value(field, value) where field is the field instance. field_value() will check the type of field using field.__class__ and based on that return a value that MongoEngine would expect. For example, the value of a normal StringField can just be returned as is, but for an EmbeddedDocumentField, an instance of that embedded document type needs to be created. It also does this trick for the items in lists fields.

from mongoengine import fields


def update_document(document, data_dict):

    def field_value(field, value):

        if field.__class__ in (fields.ListField, fields.SortedListField):
            return [
                field_value(field.field, item)
                for item in value
            ]
        if field.__class__ in (
            fields.EmbeddedDocumentField,
            fields.GenericEmbeddedDocumentField,
            fields.ReferenceField,
            fields.GenericReferenceField
        ):
            return field.document_type(**value)
        else:
            return value

    [setattr(
        document, key,
        field_value(document._fields[key], value)
    ) for key, value in data_dict.items()]

    return document

Usage:

class Pets(EmbeddedDocument):
    name = StringField()

class Person(Document):
    name = StringField()
    address = StringField()
    pets = ListField(EmbeddedDocumentField(Pets))

person = Person()

data = {
    "name": "Hank",
    "address": "Far away",
    "pets": [
        {
            "name": "Scooter"
        }
    ]
}

update_document(person, data)

Prop

@kovan What do you mean exactly? When you try to update fields that have been deleted? If you need a fallback for this, you can always write it yourself, shouldn't be too hard. – Prop

Answer 2

Pretty late to the game here, but FWIW, MongoEngine has a build in solution for this.

Regardless if you want to create or update you can do the following:

class Pets(EmbeddedDocument):
    name = StringField()

class Person(Document):
    name = StringField()
    address = StringField()
    pets = ListField(EmbeddedDocumentField(Pets))

p = Person(**{
    "name": "Hank",
    "address": "Far away",
    "pets": [{"name": "Scooter"}]
})
p.save()

Only difference for update is you need to stick in an id. That way mongoengine won't duplicate a doc with an existing id and update it instead.

Dowlen

This is the new solution. – Palmar

This solution will fail in situation when you have primary key or unique constraints in your Document model. – Atalee

Answer 3

Try something more like this

p.update(**{
    "set__name": "Hank",
    "set__address": "Far away"
})

Nerveless

Thanks for your answer. This works for normal fields but not for EmbeddedDocumentFields. I added the requirement to my question. Sorry for not being clear at once. – Prop

You can accomplish with list items index in your update query p.update(**{"set__name": "Hank", "set__address": "Far away", 'set__pets__0__name': 'Scooter'}) – Nerveless

Answer 4

I have tried most of the answers above, none of them seem really work with Embedded documents. Even though the they updated the fields they did also deleted content of unfilled fields in Embedded document.

For that I decided to take a path suggested by @hckjck, I have written a simple function that converts dict to a format so it can be processed by document.update:

def convert_dict_to_update(dictionary, roots=None, return_dict=None):
    """    
    :param dictionary: dictionary with update parameters
    :param roots: roots of nested documents - used for recursion
    :param return_dict: used for recursion
    :return: new dict
    """
    if return_dict is None:
        return_dict = {}
    if roots is None:
        roots = []

    for key, value in dictionary.iteritems():
        if isinstance(value, dict):
            roots.append(key)
            convert_dict_to_update(value, roots=roots, return_dict=return_dict)
            roots.remove(key)  # go one level down in the recursion
        else:
            if roots:
                set_key_name = 'set__{roots}__{key}'.format(
                    roots='__'.join(roots), key=key)
            else:
                set_key_name = 'set__{key}'.format(key=key)
            return_dict[set_key_name] = value

    return return_dict

Now this data:

{u'communication': {u'mobile_phone': u'2323232323', 'email':{'primary' : '[email protected]'}}}

will be converted to:

{'set__communication__mobile_phone': u'2323232323', 'set__communication__email__primary': '[email protected]'}

Which can be used like this

document.update(**conv_dict_to_update(data))

Also available in this gist: https://gist.github.com/Visgean/e536e466207bf439983a

I don't know how effective this is but it works.

Dunois

U so much helps me by this, sadly mongoengine no have user-friendly func for it – Jiggerypokery

Answer 5

Here is a function to update documents with EmbeddedDocuments. It is based upon @rednaw's solution, though accounts for EmbeddedDocuments having EmbeddedDocuments.

from mongoengine.fields import *

def field_value(field, value):
  ''' 
  Converts a supplied value to the type required by the field.
  If the field requires a EmbeddedDocument the EmbeddedDocument
  is created and updated using the supplied data.
  '''
  if field.__class__ in (ListField, SortedListField):
    # return a list of the field values 
    return [
      field_value(field.field, item) 
      for item in value]

  elif field.__class__ in (
    EmbeddedDocumentField,
    GenericEmbeddedDocumentField,
    ReferenceField,
    GenericReferenceField):

    embedded_doc = field.document_type()
    update_document(embedded_doc, value)
    return embedded_doc
  else:
    return value


def update_document(doc, data):
  ''' Update an document to match the supplied dictionary.
  '''
  for key, value in data.iteritems():

    if hasattr(doc, key):
        value = field_value(doc._fields[key], value)
        setattr(doc, key, value)
    else:
        # handle invalid key
        pass

  return doc

The key here is the field_value method updating an embedded document rather than instantiating it with the data.

Usage Example :

class Pets(EmbeddedDocument):
    name = StringField()

class Person(EmbeddedDocument):
    name = StringField()
    address = StringField()
    pets = ListField(EmbeddedDocumentField(Pets))

class Group(Document):
    name = StringField()
    members = ListField(EmbeddedDocumentField(Person))

g = Group()

update_document(g, {
  'name': 'Coding Buddies',
  'members': [
    {
      'name': 'Dawson',
      'address': 'Somewhere in Nova Scotia',
      'pets': [
        {
          'name': 'Sparkles'
        }
      ]
    },
    {
      'name': 'rednaw',
      'address': 'Not too sure?',
      'pets': [
        {
          'name': 'Fluffy'
        }
      ]
    }
  ]
})

FYI That's actually my cat's name.

EDIT : typo in variable names.

Vision

Hey Dawson, what exactly didn't work for you about my method? – Prop

@rednaw your solution did not account for embedded documents containing embedded documents. The model I am updating has a ListField of EmbeddedDocuments. These EmbeddedDocuments themselves have a ListField of EmbeddedDocuments. Your solution allowed for updating EmbeddedDocuments, ListFields, and ListFields containing EmbeddedDocuments, though not EmbeddedDocuments containing EmbeddedDocuments or ListFields. – Vision

Dawson, your usage example seems to work fine with my method. I don't understand what doesn't work for you. I tested with this script: pastebin.com/uMB0t1Xw – Prop

Not sure about this function, there are a few undefined variables... eg update_entity and entity on the very last line... – Dunois

That was a type. I fixed this in an edit. entity was supposed to be a document. – Vision

Answer 6

You have to use the from_json method of Document class. Refer this. It works for EmbeddedDocumentField as well.

In your case, you have to do this

import json

p = Person.from_json(json.dumps({
    "name": "Hank",
    "address": "Far away",
    "pets": [
        {
            "name": "Scooter"
        }
    ]
}))

Remember, this is still not saved. You can save it by adding p.save()

Note: The from_json method has additional parameters to choose whether you want to create a brand new document or update an existing. Do look at the documentation linked in this answer.

Crump

Answer 7

To store python dict as a sub document one may use mongoengine.fields.DictField.

Checkout manuals.

A dictionary field that wraps a standard Python dictionary. This is similar to an embedded document, but the structure is not defined.

Citriculture