Google App Engine bulkloader issue when using yaml autogenerated configuration and entities with numeric ID

Question

My application uses Django non-rel. I don't have access to model.

I have my bulkloader.yaml file autogenerated by appcfg.py create_bulkloader_config.

Problem is entities numeric ID's are being imported as string key names. So if I export entity with int ID of, for example, '62', it gets imported as entity with string key name of '61' which screws up Django.

Revelant bulkloader.yaml Fragment:

 property_map:
  - property: __key__
    external_name: key
    export_transform: transform.key_id_or_name_as_string

I'm trying to setup download/upload od data using bulkloader, and I want to have data as easy to understand format (like .csv) --- so using bulkloader.py --dump (...) is not a viable option since it gives me sqlite3 files that have entities contents pickled as a single row.

EDIT

I tried doing what @Nick suggested and I got an exception:

 ErrorOnTransform: Numeric keys are not supported on input at this time.

Does this mean that I have to stick to bulkloader.py (that uses that werid sqlite format) or I messed something? ;)

Header of Transformer:

- kind: auth_user
    connector: csv
    connector_options:
      encoding: utf-8
      skip_import_header_row: True
      print_export_header_row: True

    property_map:
      - property: __key__
        external_name: key
        export_transform: transform.key_id_or_name_as_string
        import_transform: transform.create_foreign_key('auth_user', key_is_id=True) 

Whole Stacktrace:

      Traceback (most recent call last):
      File "/opt/google/appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
        status, instruction = item.PerformWork(self.__thread_pool)
      File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork
        transfer_time = self._TransferItem(thread_pool)
      File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem
        self.content = self.request_manager.EncodeContent(self.rows)
      File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent
        entity = loader.create_entity(values, key_name=key, parent=parent)
      File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity
        return self.dict_to_entity(input_dict, self.bulkload_state)
      File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 131, in dict_to_entity
        instance = self.__create_instance(input_dict, bulkload_state_copy)
      File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 209, in __create_instance
        'Numeric keys are not supported on input at this time.')

Twocycle

Answer 1

You've got the export_transform 'key_id_or_name_as_string', but you don't have a corresponding import transform. Try this:

property_map:
 - property: __key__
   external_name: key
   export_transform: transform.key_id_or_name_as_string
   import_transform: transform.create_foreign_key('Kind', key_is_id=True)

Where 'Kind' is the name of the kind identified in the config file.

Kamikamikaze

Thanks for answer. My problem is with primary key, is just the name of function create_foreign_key misleading? – Twocycle

The name is misleading - it should probably be simply 'create_key'. – Kamikamikaze

Well I get an exception. . . that numeric keys are unsupported. I updated the question. – Twocycle

Answer 2

This works fine for me:

import_transform: transform.none_if_empty(long)

Syncretism