AppEngine bulkloader export Model with self-defined Property

Question

I want to use bulkloader to download all entities in a model with some self-defined Property.

If I define a model like this,

class MyType:
    def __init__(self, arg):
        self.name = arg['name']
        self.id = arg['id']

class MyProperty(db.Property):
    def get_value_for_datastore(self, instance):
        val = super(MyProperty, self).get_value_for_datastore(instance)
        if type(val) == dict:
            val = MyType(val)
        return pickle.dumps(val)

    def make_value_from_datastore(self, val):
        return None if val is None else pickle.loads(str(val))

class MyModel(db.Model):
    info = MyProperty()

then how can I download MyModel using the bulkloader such that there will not be un-pickled value in the file? I think I should define the export_transform for info in bulkloader.yaml, but I don't know what it should be like.

transformers:
- kind: MyModel
  connector: csv
  property_map:
  - property: __key__
    external_name: log_id
    export_transform: transform.key_id_or_name_as_string
  - property: info
    external_name: info
    export_transform: ### HERE ###

I've seen transform.py but still have no idea about how it works. Please tell my any method that can solve my problem. Thanks.

Highpressure

What do you mean "such that there will not be un-pickled value in the file"? Do you not want that field to show up, or do you want it there pickled, or something else? – Joung

What I want to say is, I want that field to show in the CSV, but I want it to be unpickled before being written to the CSV. Currently the field will show in the CSV as a pickled value. – Highpressure

An unpickled string is an object. You can't output an object to a text file, only a string representation of it. What representation do you want to write to the file? – Joung

If using the above example, I want to know if I can write info.name or info.name,info.id to the field "info" in the CSV file. – Highpressure

Answer 1

Okay I'm answering my own questions...

I still don't know why pickle doesn't work, but after changing to use simplejson instead of pickle, I can successfully export MyProperty in the designated format.

The bulkloader.yaml may look like this.

python_preamable:
- import myutils
- import django.utils.simplejson
...

transformers:
- kind: MyModel
  connector: csv
  property_map:
  ...
  - property: info
    external_name: info
    export_transform: myutils.load_info

And in myutils.py load_info may look like this.

def load_info():
    def load(x):
        if not x:
            return ''
        info = simplejson.loads(x)
        return '%s-%s' % (info['id'], info['name']) # the output format for info
    return load

Highpressure