Google app engine bulkloader opens empty file at upload

Question

I'm running the following two commands:

appcfg.py download_data --kind=Journal --url=http://appname.appspot.com/_ah/remote_api --filename=Journals.sql3
appcfg.py upload_data --url=http://localhost:8080/_ah/remote_api --kind=Journal --filename=Journals.sql3

The first one successfully downloads the data in the following files: bulkloader-results-20130718.112127.sql3 Journals.sql3 No errors whatsoever. When I open the file in notepad, it looks like the data is in there.

The second one opens a random new empty file each time I run it (e.g. bulkloader-progress-20130718.115805.sql3). Then it gives the following error log:

INFO     2013-07-18 11:54:48,930 module.py:595] default: "POST /_ah/remote_api HTTP/1.1" 200 96
INFO     2013-07-18 11:58:17,563 module.py:595] default: "GET /_ah/remote_api?rtok=36123921049 HTTP/1.1" 200 64
INFO     2013-07-18 11:58:20,609 module.py:595] default: "GET /_ah/remote_api HTTP/1.1" 200 37
ERROR    2013-07-18 09:58:20,618 handler.py:384] Exception while handling service_name: "datastore_v4"
method: "AllocateIds"
request: ""
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remote_api\handler.py", line 380, in post
    response_data = self.ExecuteRequest(request)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remote_api\handler.py", line 411, in ExecuteRequest
    response_data)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 328, in MakeSyncCall
    rpc.CheckSuccess()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
    self.request, self.response)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 234, in _MakeRealSyncCall
    raise pickle.loads(response_pb.exception())
AttributeError: 'NoneType' object has no attribute 'THREADSAFE'
INFO     2013-07-18 11:58:20,622 module.py:595] default: "POST /_ah/remote_api HTTP/1.1" 200 96

What am I doing wrong?

EDIT: Still no solution. What is funny is that uploading on the app engine server does work.

Crossover

The random named filesare progress files, you can name progress files explicitly which allows you to restart a halted upload or download. They are not your problem. – Smashing

Ok, thanks a lot for clearing that out. Any idea what could cause the error? – Crossover

Unfortunately not, what SDK version and python version are you using ? – Smashing

python.org/download/releases/2.7.5 and developers.google.com/appengine/… on Windows 8 (just downloaded yesterday, as I have a new laptop) – Crossover

Also removing '--kind=Journal' does not work. – Crossover

The problem as the error states is with AllocateIds this service is used when uploading to ensure the same id's are allocated to the uploaded entities. Is the local dev datastore empty ? Also do you have auto_id_policy set in your app.yaml ? – Smashing

It's empty in terms of Journals, but there's a few other objects in there already. There's no auto_id_policy set. I didn't set this policy in my previous development environment. – Crossover

Tried emptying the datastore & setting the auto_id_policy. Nothing works.. Any ideas? – Crossover

Also tried turning threadsafe off, no effect. – Crossover

I have the same problem here. I recently installed the latest SDK version 1.8.2. How about you? – Consistent

Yes, I have the newest one also. – Crossover

Answer 1

Looks like there's a ticket for this issue, and it's related to the latest version of the Python dev server (1.8.2).

As suggested on the ticket, I downgraded my dev server to the previous SDK version (1.8.1) and bulk upload worked as expected again.

Mannino

new version 1.8.3 has the same problem. – Osvaldooswal