Appengine BulkLoader problem using development server

Question

I am doing something that should be simple and getting stuck over and over again. I am getting this error when uploading quite simple csv files to my development server:

Error in WorkerThread-0: app "dev~fbdec" cannot access app "fbdec"'s data

The exact output being:

[INFO    ] Logging to bulkloader-log-20110815.142554
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20110815.142554.sql3
Please enter login credentials for localhost
Email: [email protected]
Password for [email protected]: 
[INFO    ] Connecting to localhost:8080/remote_api
[INFO    ] Skipping header line.
[INFO    ] Starting import; maximum 10 entities per post
[ERROR   ] [WorkerThread-0] WorkerThread:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 176, in WorkOnItems
 status, instruction = item.PerformWork(self.__thread_pool)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 764, in PerformWork
transfer_time = self._TransferItem(thread_pool)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 935, in _TransferItem
self.request_manager.PostEntities(self.content)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1418, in PostEntities
datastore.Put(entities)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 467, in Put
return PutAsync(entities, **kwargs).get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 658, in get_result
results = self.__rpcs[0].get_result()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1385, in __put_hook
self.check_rpc_success(rpc)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1074, in check_rpc_success
raise _ToDatastoreError(err)
BadRequestError: app "dev~fbdec" cannot access app "fbdec"'s data
[INFO    ] An error occurred. Shutting down...
[ERROR   ] Error in WorkerThread-0: app "dev~fbdec" cannot access app "fbdec"'s data
[INFO    ] 2 entities total, 0 previously transferred
[INFO    ] 0 entities (723 bytes) transferred in 7.9 seconds
[INFO    ] Some entities not successfully transferred

This is the app.yaml file:

application: fbdec
version: 1
runtime: python
api_version: 1

handlers:
- url: /static
  static_dir: static

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

- url: /.*
  script: fbdec.py

This the shell call to perform the upload:

appcfg.py upload_data --config_file=fbdec/fbuploader.py --filename=cols.csv --kind=Cols --has_header --num_threads=1 --url=http://localhost:8080/remote_api fbdec/

This is the uploader code:

import cgi
import datetime
import logging
import os

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader
from os import environ


class Grobs(db.Model):
    coleccion = db.TextProperty()
    objeto= db.TextProperty() 
    descripcion = db.TextProperty()

class GrobLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Grobs',
                                   [('coleccion', str),  
                                    ('objeto', str),
                                    ('descripcion', str)
                                   ])

class Cols(db.Model):
    coleccion = db.TextProperty()
    descripcion= db.TextProperty() 

class ColLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Cols',
                                   [('coleccion', str),
                                    ('descripcion', str)
                                   ])

loaders = [GrobLoader, ColLoader]

And this the cols.csv file:

coleccion,descripcion
gafas,descripcion
sombreros,descripcion

Any help will be really welcome. I cant figure out what I am doing wrong.

I am using Appengine 1.5.2 on Mac Osx Leopard. I have tried both both python 2.5.4 and 2.6.6 (just in case).

Thanks in advance and best regards !

Econah

The issue is that to help expose problems apps would have with the "s~" prefix on High Redundancy application names, a "dev~" prefix was added to development server applications. In theory, when everything's working right, these prefixes should be transparent. You may need to supply --application= on the command line, although I'm not certain whether you want to include the dev~ or not there. – Capitulary

Thanks for the explanation. Didnt know that. I have tried Robert Kluin´s parameter and works fine too. Best. – Econah

Upgrading to the latest SDK should fix this. – Nay

Its up to date. In fact it is the other way round. I have rolled back to an older version (a 1.3.5 Installer I had somewhere) to test and there the problem doesnt arise and it works without the --default_partition option added. – Econah

Answer 1

When starting dev_appserver, pass the --default_partition="" parameter.

Bidget

Thanks VERY much. That solved the problem. Thanks for the amazingly quick response. All the best. – Econah

Javier; use the tick to accept the answer, which closes the question and makes Robert a tiny bit happier inside. – Installation

To do this on GoogleAppEngineLauncher, right click on the app then "Info..." from the menu and for "extra flags" type --default_partition= and update. Specifically note that the quotes at the end should not be added. – Hudibrastic

I am having a similar problem with python 2.7.3. Except the particular exception is No module named my_model. Details here: #13778105 – Damper

Apparently, the appcfg.py in SDK v1.8.6 doesn't recognize this flag. Any other ideas? – Metatarsal

Answer 2

On newer SDK, use the --application=dev~your-app option.

Brunner

Answer 3

I recently had the same problem, hence I created a script to do this:

APP=$1 #your app name
IN=$2 #the file containing the data to upload
DIR=$3 #the folder containing your app.yaml
appcfg.py upload_data [email protected] --passin --application=dev~$APP --filename=$IN --url=http://localhost:8080/_ah/remote_api $DIR -v

Haplosis