Play: How to transform JSON while writing/reading it to/from MongoDB

Question

Here is an simple JSON I want to write/read to/from MongoDB:

{
  "id": "ff59ab34cc59ff59ab34cc59",
  "name": "Joe",
  "surname": "Cocker"
}

Before storing it in MongoDB, "ff59ab34cc59ff59ab34cc59" has to be transformed into an ObjectID and id renamed to _id... so given the following Reads, how do I achieve that?

val personReads: Reads[JsObject] = (
  (__ \ 'id).read[String] ~ // how do I rename id to _id AND transform "ff59ab34cc59ff59ab34cc59" to an ObjectID?
  (__ \ 'name).read[String] ~
  (__ \ 'surname).read[String]
) reduce

And of course, I also need the contrary for my Writes, i.e. renaming _id to id and transforming an ObjectID to plain text in the format "ff59ab34cc59ff59ab34cc59".

Galligaskins

Is there any reason you can't read and write your documents with "_id" instead of "id"? Also, why do you want to convert id to an ObjectID type? You can store any type in the _id field in MongoDB. – Bostick

"_id" is a MongoDB technicality... and I don't want to introduce dependencies like this in my model. Then, of course, you can store any type you want in the _id field... but an ObjectID just requires 12 bytes to store the [gu]id (in binary), while a plain text representation requires 24 bytes. – Galligaskins

What language are you using for your application? Have you considered using an ODM layer? – Bostick

I'm using Scala... and I'm using ReactiveMongo to deal with MongoDB. – Galligaskins

Answer 1

JsonExtensions

I usually have a JsExtensions object in my application which looks like the following :

import reactivemongo.bson.BSONObjectID
object JsonExtensions {

  import play.api.libs.json._

  def withDefault[A](key: String, default: A)(implicit writes: Writes[A]) = __.json.update((__ \ key).json.copyFrom((__ \ key).json.pick orElse Reads.pure(Json.toJson(default))))
  def copyKey(fromPath: JsPath,toPath:JsPath ) = __.json.update(toPath.json.copyFrom(fromPath.json.pick))
  def copyOptKey(fromPath: JsPath,toPath:JsPath ) = __.json.update(toPath.json.copyFrom(fromPath.json.pick orElse Reads.pure(JsNull)))
  def moveKey(fromPath:JsPath, toPath:JsPath) =(json:JsValue)=> json.transform(copyKey(fromPath,toPath) andThen fromPath.json.prune).get
}

For a simple model

case class SOUser(name:String,_id:BSONObjectID)

you can write your json serializer/deserializer like this:

object SOUser{
  import play.api.libs.json.Format
  import play.api.libs.json.Json
  import play.modules.reactivemongo.json.BSONFormats._

  implicit val soUserFormat= new Format[SOUser]{
    import play.api.libs.json.{JsPath, JsResult, JsValue}
    import JsonExtensions._
    val base = Json.format[SOUser]
    private val publicIdPath: JsPath = JsPath \ 'id
    private val privateIdPath: JsPath = JsPath \ '_id \ '$oid

    def reads(json: JsValue): JsResult[SOUser] = base.compose(copyKey(publicIdPath, privateIdPath)).reads(json)
    def writes(o: SOUser): JsValue = base.transform(moveKey(privateIdPath,publicIdPath)).writes(o)
  }
}

here is what you get in the console :

scala> import reactivemongo.bson.BSONObjectID
import reactivemongo.bson.BSONObjectID

scala> import models.SOUser
import models.SOUser

scala> import play.api.libs.json.Json
import play.api.libs.json.Json

scala>

scala> val user = SOUser("John Smith", BSONObjectID.generate)
user: models.SOUser = SOUser(John Smith,BSONObjectID("52d00fd5c912c061007a28d1"))

scala> val jsonUser=Json.toJson(user)
jsonUser: play.api.libs.json.JsValue = {"name":"John Smith","id":"52d00fd5c912c061007a28d1","_id":{}}

scala> Json.prettyPrint(jsonUser)
res0: String =
{
  "name" : "John Smith",
  "id" : "52d00fd5c912c061007a28d1",
  "_id" : { }
}

scala> jsonUser.validate[SOUser]
res1: play.api.libs.json.JsResult[models.SOUser] = JsSuccess(SOUser(John Smith,BSONObjectID("52d00fd5c912c061007a28d1")),/id)

Applying this to your example

val _personReads: Reads[JsObject] = (
  (__ \ 'id).read[String] ~
  (__ \ 'name).read[String] ~
  (__ \ 'surname).read[String]
).reduce

Doesn't compile by default, I guess you meant to write :

val _personReads: Reads[(String,String,String)] = (
  (__ \ 'id).read[String] ~
  (__ \ 'name).read[String] ~
  (__ \ 'surname).read[String]
).tupled

in which case you can do the following

import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._
import play.modules.reactivemongo.json.BSONFormats._
import reactivemongo.bson.BSONObjectID

def copyKey(fromPath: JsPath,toPath:JsPath ) = __.json.update(toPath.json.copyFrom(fromPath.json.pick))

val json = """{
  "id": "ff59ab34cc59ff59ab34cc59",
  "name": "Joe",
  "surname": "Cocker"
}"""

val originaljson = Json.parse(json)
val publicIdPath: JsPath = JsPath \ 'id
val privateIdPath: JsPath = JsPath \ '_id \ '$oid

val _personReads: Reads[(BSONObjectID,String,String)] = (
  (__ \ '_id).read[BSONObjectID] ~
  (__ \ 'name).read[String] ~
  (__ \ 'surname).read[String]
).tupled
val personReads=_personReads.compose(copyKey(publicIdPath,privateIdPath))

originaljson.validate(personReads)
// yields res5: play.api.libs.json.JsResult[(reactivemongo.bson.BSONObjectID, String, String)] = JsSuccess((BSONObjectID("ff59ab34cc59ff59ab34cc59"),Joe,Cocker),/id)

or you meant that you want to move the value of the id key to _id \ $oid which can be accomplished with

import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._
import play.modules.reactivemongo.json.BSONFormats._
import reactivemongo.bson.BSONObjectID

def copyKey(fromPath: JsPath,toPath:JsPath ) = __.json.update(toPath.json.copyFrom(fromPath.json.pick))

val json = """{
  "id": "ff59ab34cc59ff59ab34cc59",
  "name": "Joe",
  "surname": "Cocker"
}"""

val originaljson = Json.parse(json)
val publicIdPath: JsPath = JsPath \ 'id
val privateIdPath: JsPath = JsPath \ '_id \ '$oid

originaljson.transform(copyKey(publicIdPath,privateIdPath) andThen publicIdPath.json.prune)

You can't have a BSONObjectID in there for now since you are manipulating object from the JsValue type hierarchy. When you pass json to reactivemongo it is converted to a BSONValue. A JsObject will be converted to a BSONDocument. if the JsObject contains a path for _id\$oid this path will be converted to a BSONObjectId automatically and it will be stored as an ObjectID in mongodb.

Builtin

Answer 2

The original question is really about reactivemongo's (sgodbillon et al) treatment of the native mongodb _id. The chosen answer is instructive and correct but obliquely addresses the OP's concern whether "it will all just work".

Thanks to https://github.com/ReactiveMongo/ReactiveMongo-Play-Json/blob/e67e507ecf2be48cc71e429919f7642ea421642c/src/main/scala/package.scala#L241-L255, I believe it will.

import scala.concurrent.Await
import scala.concurrent.duration.Duration

import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.libs.functional.syntax._
import play.api.libs.json._
import play.modules.reactivemongo.json.collection.JSONCollection
import reactivemongo.api._
import reactivemongo.bson.BSONObjectID
import reactivemongo.play.json._

case class Person(
id: BSONObjectID,
name: String,
surname: String
)

implicit val PersonFormat: OFormat[Person] = (
  (__ \ "_id").format[BSONObjectID] and
    (__ \ "name").format[String] and
    (__ \ "surname").format[String]
)(Person.apply, unlift(Person.unapply))

val driver = new reactivemongo.api.MongoDriver
val connection = driver.connection(List("localhost"))
val db = connection.db("test")
val coll = db.collection[JSONCollection]("persons")
coll.drop(false)

val id = BSONObjectID.generate()
Await.ready(coll.insert(Person(id, "Joe", "Cocker")), Duration.Inf)
Await.ready(coll.find(Json.obj()).one[Person] map { op => assert(op.get.id == id, {}) }, Duration.Inf)

The above is a minimal working example of your case class using id and the database storing it as _id. Both are instantiated as 12-byte BSONObjectIDs.

Pucka