Copy an entity in Google App Engine datastore in Python without knowing property names at 'compile' time

Question

In a Python Google App Engine app I'm writing, I have an entity stored in the datastore that I need to retrieve, make an exact copy of it (with the exception of the key), and then put this entity back in.

How should I do this? In particular, are there any caveats or tricks I need to be aware of when doing this so that I get a copy of the sort I expect and not something else.

ETA: Well, I tried it out and I did run into problems. I would like to make my copy in such a way that I don't have to know the names of the properties when I write the code. My thinking was to do this:

#theThing = a particular entity we pull from the datastore with model Thing
copyThing = Thing(user = user)
for thingProperty in theThing.properties():
    copyThing.__setattr__(thingProperty[0], thingProperty[1])

This executes without any errors... until I try to pull copyThing from the datastore, at which point I discover that all of the properties are set to None (with the exception of the user and key, obviously). So clearly this code is doing something, since it's replacing the defaults with None (all of the properties have a default value set), but not at all what I want. Suggestions?

Smalley

Answer 1

Here you go:

def clone_entity(e, **extra_args):
  """Clones an entity, adding or overriding constructor attributes.

  The cloned entity will have exactly the same property values as the original
  entity, except where overridden. By default it will have no parent entity or
  key name, unless supplied.

  Args:
    e: The entity to clone
    extra_args: Keyword arguments to override from the cloned entity and pass
      to the constructor.
  Returns:
    A cloned, possibly modified, copy of entity e.
  """
  klass = e.__class__
  props = dict((k, v.__get__(e, klass)) for k, v in klass.properties().iteritems())
  props.update(extra_args)
  return klass(**props)

Example usage:

b = clone_entity(a)
c = clone_entity(a, key_name='foo')
d = clone_entity(a, parent=a.key().parent())

EDIT: Changes if using NDB

Combining Gus' comment below with a fix for properties that specify a different datastore name, the following code works for NDB:

def clone_entity(e, **extra_args):
  klass = e.__class__
  props = dict((v._code_name, v.__get__(e, klass)) for v in klass._properties.itervalues() if type(v) is not ndb.ComputedProperty)
  props.update(extra_args)
  return klass(**props)

Example usage (note key_name becomes id in NDB):

b = clone_entity(a, id='new_id_here')

Side note: see the use of _code_name to get the Python-friendly property name. Without this, a property like name = ndb.StringProperty('n') would cause the model constructor to raise an AttributeError: type object 'foo' has no attribute 'n'.

Golub

This isn't working for me: AttributeError: type object 'NoneType' has no attribute 'properties' – Bioplasm

@Rosarch It sounds like you're passing in None to the clone_entity method. Naturally, you can't clone 'None'. – Golub

Nice and clean, but it will pull reference properties from the datastore if needed. e.g. if the entity has 20 ReferenceProperty fields, then they will each be pulled from the datastore sequentially. (20 round trips is bad :) This can be fixed by checking for db.ReferenceProperty and using get_value_for_datastore. – Assuming

Be sure to manually set any db.DateTimeProperty with auto_now_add=True, since cloning will copy the original entity's created datetime. (unless that's what you want to do) – Assuming

Hi, Could anyone help provide details how to execute above code (given by Nick) on local development server? Are below steps correct? - create py file for function clone_entity - upload to dev appserver using appcfg.py - run b = clone_entity(a) in interactive console. If steps are correct, appreciate if anyone could help provide detailed command. Thanks! – Sabrasabre

@user580825 It's a Python function - you use it as you would any other Python function. If you want to use it in the interactive console, you can just paste it right in there. – Golub

For the next generations using NDB, you will have to change klass.properties() for klass._properties. And you'll get an error for Computed properties so watch out for that. I replaced the props var for this: props = dict((k, v.__get__(e, klass)) for k, v in klass._properties.iteritems() if type(v) is not ndb.ComputedProperty) – Unison

AttributeError: type object 'mydescendantofndbModel' has no attribute 'properties' – Extracanonical

For models containing Structured properties, you may want to make a deep copy of those by adding: if type(v) is ndb.StructuredProperty: props.append((v._code_name, copy.deepcopy(v.__get__(e, klass)))) – Budd

NOTE: if your model is an Expando you would have to provide the desired expando property/values in **extra_args else they won't be copied – Lied

@Budd Since props is a dictionary, it would have to be an indexed assignment: props[v._code_name] = copy.deepcopy(v.__get__(e, klass)) – Mcdavid

If it helps anyone, when cloning a PolyModel one of the props is 'class_' with a list of the class and parent class. To use this function you need to remove it before the new entity is returned. ` if 'class_' in props: props.pop('class_')` – Trimurti

Answer 2

If you're using the NDB you can simply copy with: new_entity.populate(**old_entity.to_dict())

Malignant

How would I modify the clone's key name? – Atrium

to_dict() returns what is in the _properties list... which may include properties that don't exist on the model class (were removed). this will cause an error "type object '[your model name]' has no attribute 'a property that was deleted'" – Lied

Answer 3

This is just an extension to Nick Johnson's excellent code to address the problems highlighted by Amir in the comments:

1. The db.Key value of the ReferenceProperty is no longer retrieved via an unnecessary roundtrip to the datastore.
2. You can now specify whether you want to skip DateTime properties with the auto_now and/or auto_now_add flag.

Here's the updated code:

def clone_entity(e, skip_auto_now=False, skip_auto_now_add=False, **extra_args):
  """Clones an entity, adding or overriding constructor attributes.

  The cloned entity will have exactly the same property values as the original
  entity, except where overridden. By default it will have no parent entity or
  key name, unless supplied.

  Args:
    e: The entity to clone
    skip_auto_now: If True then all DateTimeProperty propertes will be skipped which have the 'auto_now' flag set to True
    skip_auto_now_add: If True then all DateTimeProperty propertes will be skipped which have the 'auto_now_add' flag set to True
    extra_args: Keyword arguments to override from the cloned entity and pass
      to the constructor.
  Returns:
    A cloned, possibly modified, copy of entity e.
  """

  klass = e.__class__
  props = {}
  for k, v in klass.properties().iteritems():
    if not (type(v) == db.DateTimeProperty and ((skip_auto_now and getattr(v, 'auto_now')) or (skip_auto_now_add and getattr(v, 'auto_now_add')))):
      if type(v) == db.ReferenceProperty:
        value = getattr(klass, k).get_value_for_datastore(e)
      else:
        value = v.__get__(e, klass)
      props[k] = value
  props.update(extra_args)
  return klass(**props)

The first if expression is not very elegant so I appreciate if you can share a better way to write it.

Foothill

Thanks for writing this. It's exactly what I was looking for. It does need one change; there is a reference to self even though this isn't an object method. The line value = v.__get__(self, klass) needs to change to value = v.__get__(e, klass) – Lenticel

Opps, I missed a second reference to self. value = getattr(klass, k).get_value_for_datastore(self) should become value = getattr(klass, k).get_value_for_datastore(e) – Lenticel

Corrected. Thanks and sorry! – Foothill

I was wondering about why not just using _properties.iteritems, as the resulting keys are just the property names. – Mcdavid

Answer 4

I'm neither Python nor AppEngine guru, but couldn't one dynamically get/set the properties?

props = {}
for p in Thing.properties():
    props[p] = getattr(old_thing, p)
new_thing = Thing(**props).put()

Palestrina

(Self note: read the entire question before answering, not just the topic.) – Palestrina

Answer 5

A variation inspired in Nick's answer which handles the case in which your entity has a (repeated) StructuredProperty, where the StructuredProperty itself has ComputedProperties. It can probably be written more tersely with dict comprehension somehow, but here is the longer version that worked for me:

def removeComputedProps(klass,oldDicc):
  dicc = {}
  for key,propertType in klass._properties.iteritems():
      if type(propertType) is ndb.StructuredProperty:
          purged = []
          for item in oldDicc[key]:
              purged.append(removeComputedProps(propertType._modelclass,item))
          dicc[key]=purged
      else:
          if type(propertType) is not ndb.ComputedProperty:
              dicc[key] = oldDicc[key]
  return dicc

def cloneEntity(entity):
  oldDicc = entity.to_dict() 
  klass = entity.__class__
  dicc = removeComputedProps(klass,oldDicc)
  return klass(**dicc)

Voleta

Answer 6

This can be tricky if you've renamed the underlying keys for your properties... which some people opt to do instead of making mass data changes

say you started with this:

class Person(ndb.Model):
   fname = ndb.StringProperty()
   lname = ndb.StringProperty()

then one day you really decided that it would be nicer to use first_name and last_name instead... so you do this:

class Person(ndb.Model):
   first_name = ndb.StringProperty(name="fname")
   last_name = ndb.StringProperty(name="lname")

now when you do Person._properties (or .properties() or person_instance._properties) you will get a dictionary with keys that match the underlying names (fname and lname)... but won't match the actual property names on the class... so it won't work if you put them into the constructor of a new instance, or use the .populate() method (the above examples will break)

In NDB anyways, instances of models have ._values dictionary which is keyed by the underlying property names... and you can update it directly. I ended up with something like this:

    def clone(entity, **extra_args):
        klass = entity.__class__
        clone = klass(**extra_args)
        original_values = dict((k,v) for k,v in entity._values.iteritems() if k not in clone._values)
        clone._values.update(original_values)
        return clone

This isn't really the safest way... as there are other private helper methods that do more work (like validation and conversion of computed properties by using _store_value() and _retrieve_value())... but if you're models are simple enough, and you like living on the edge :)

Lied

Answer 7

Here's the code provided by @zengabor with the if expression formatted for easier reading. It may not be PEP-8 compliant:

klass = e.__class__
props = {}
for k, v in klass.properties().iteritems():
    if not (type(v) == db.DateTimeProperty and ((
            skip_auto_now     and getattr(v, 'auto_now'    )) or (
            skip_auto_now_add and getattr(v, 'auto_now_add')))):
        if type(v) == db.ReferenceProperty:
            value = getattr(klass, k).get_value_for_datastore(e)
        else:
            value = v.__get__(e, klass)
        props[k] = value
props.update(extra_args)
return klass(**props)

Mcdavid