ORM for google cloud datastore

Question

https://developers.google.com/datastore/docs/overview

It looks like datastore in GAE but without ORM (object relation model). May I used the same ORM model as datastore on GAE for Cloud Datastore? or Is there any ORM support can be found for Cloud Datastore?

Backwards

Not an orm but this might be usefull: github.com/transceptor-technology/dbWrapper – Auten

github.com/lucemia/google-cloud-datastore-orm an orm approach – Backwards

Answer 1

Google Cloud Datastore only provides a low-level API (proto and json) to send datastore RPCs.

NDB and similar higher level libraries could be adapted to use a lower level wrapper like googledatastore (reference) instead of google.appengine.datastore.datastore_rpc

Smoke

Thanks, Is there any project for that already? (I don't mind if I need to create it myself, I just want to know before I start :) – Backwards

@Backwards not that I know of. – Smoke

@Backwards you could also consider contributing your changes directly to ndb: code.google.com/p/appengine-ndb-experiment/wiki/Contributing – Smoke

@Backwards do you have an open source repo where you're working on the ndb implementation? – Posthumous

Someone, who managed to use cloud datastore with NDB, please supply some hint! – Cousins

I face some truoble while trying the datastore_rpc approach, thus I create a datastore_stub. github.com/lucemia/google-cloud-datastore-orm – Backwards

Answer 2

App Engine Datastore high level APIs, both first party (db, ndb) and third party (objectify, slim3), are built on top of low level APIs:

• datastore_rpc for Python
• DatastoreService/AsyncDatastoreService for Java

Replacing the App Engine specific versions of these interfaces/classes to work on top of The Google Cloud Datastore API will allow you to use these high level APIs outside of App Engine.

The high level API code itself should not have to change (much).

Crosshead

Answer 3

For the .NET folks out there, I just created pogo, which is a .NET ORM that supports saving POCOs to Google Cloud Datastore, and querying them using LINQ.

It is available on nuget, called "pogo", and the source is hosted here - http://code.thecodeprose.com/pogo

For example, here's an insert and a lookup:

var unique = Guid.NewGuid().ToString();

var poco = new TestDepartment
{
    Id = unique,
    Code = 123,
    Director = new TestEmployee { FirstName = "Boss" }
};

using (var session = _datastore.OpenSession())
{
    session.Store(poco);
    session.SaveChanges();
    var lookupPoco = session.Load<TestDepartment>(unique).SingleOrDefault();

    Assert.AreEqual("Boss", lookupPoco.Director.FirstName);
}

and here's a query:

using (var session = _datastore.OpenSession())
{
    var results = session.Query<TestEmployee>()
          .Where(t => t.HourlyRate > 50.0)
          .ToList();

    Assert.IsTrue(results.Count > 0);
}

Reld

Answer 4

This is not an exact answer and We already know Google is working for NDB Library, I couldn't wait for that.

What I tried is to write NDB Properties not listed in datastore_v1_pb2.py, such as GeoPt.

class GCDFoo(ndb.Model):
    latlng = ndb.GeoPtProperty()

in this case, if we read the entity by GCD lowlevel API, returns like following.

name: "latlng"
value {
  entity_value {
    property {
      name: "x"
      value {
        double_value: 10.0
        indexed: false
      }
    }
    property {
      name: "y"
      value {
        double_value: 10.0
        indexed: false
      }
    }
  }
  meaning: 9
}

hm, I don't really know what does 'meaning' mean, but It was important to describe GeoPt. and now We can write GeoPt property something like this.

def make_geopt_value(lat,lng):
  entity = datastore.Entity()
  prop = entity.property.add()
  datastore.helper.set_property(prop,'x',lng,indexed=False)
  prop = entity.property.add()
  datastore.helper.set_property(prop,'y',lat,indexed=False)
  value = datastore.Value()
  datastore.helper.set_value(value,entity)
  value.meaning = 9
  return value

It worked for me, still don't know if it's right approach though. Anyway I hope my answer is helpful for someone who can't wait NDB Library.

Dole

Answer 5

You can take a look on this one, this one is written in Typescript

Description: ts-datastore-orm targets to provide a strong typed and structural ORM feature for Datastore (Firestore in Datastore mode).

https://www.npmjs.com/package/ts-datastore-orm

import {BaseEntity, Batcher, Column, datastoreOrm, Entity, Transaction, DatastoreOrmDatastoreError, DatastoreOrmError} from "ts-datastore-orm";

@Entity({namespace: "testing", kind: "user", compositeIndexes: [{id: "desc"}, {string: "asc", ["object.name"]: "asc", __ancestor__: false}]})
export class User extends BaseEntity {
    @Column({generateId: true})
    public id: number = 0;

    @Column()
    public date: Date = new Date();

    @Column({index: true})
    public string: string = "";

    @Column()
    public number: number = 10;

    @Column()
    public buffer: Buffer = Buffer.alloc(1);

    @Column()
    public array: number[] = [];

    @Column({excludeFromIndexes: ["object.name"]})
    public object: any = {};

    @Column()
    public undefined: undefined = undefined;

    @Column()
    public null: null = null;
}


async function queryExamples() {
    const [user1, requestResponse1] = await User.query().runOnce();
    const [userList1, requestResponse2] = await User.findMany([1, 2, 3, 4, 5]);
    const [userList2, requestResponse3] = await User.findMany({ancestor: user1, ids: [1, 2, 3, 4, 5]});
    const [userList3, requestResponse4] = await User.query().run();

    const user2 = User.create({id: 1});
    const query = TaskGroup.query()
        .filter("number", "=", 10)
        .filterAny("anyColumn.name", ">", 5)
        .setAncestor(user2)
        .groupByAny("anyColumn.name")
        .orderAny("anyColumn.name", {descending: true})
        .offset(5)
        .limit(10);

    while (query.hasNextPage()) {
        const [entities] = await query.run();
    }

    // stream
    const stream = query.runStream()
        .on("data", (entity) => {
            //
        })
        .on("info", (info) => {
            //
        })
        .on("error", (error) => {
            //
        })
        .on("end", () => {
            //
        });
}

Trousseau

Answer 6

Yes. Not only can you use the same ORM models, Google Cloud Datastore allows you to read and write from your current App Engine apps storage.

https://developers.google.com/datastore/docs/activate?hl=en#google_cloud_datastore_for_an_existing_app_engine_application

Galina

please provider more details. I only saw the instruction about how to connect cloud datastore on app engine. Nothing about the ORM. – Backwards

ORM as in the features a layer like ndb provides? – Galina