What `JObject(rec) <- someJArray` means inside for-comprehension

Question

I'm learning Json4s library.

I have a json fragment like this:

{
    "records":[
        {
            "name":"John Derp",
            "address":"Jem Street 21"
        },
        {
            "name":"Scala Jo",
            "address":"in my sweet dream"
        }
    ]
}

And, I have Scala code, which converts a json string into a List of Maps, like this:

import org.json4s._
import org.json4s.JsonAST._
import org.json4s.native.JsonParser

  val json = JsonParser.parse( """{"records":[{"name":"John Derp","address":"Jem Street 21"},{"name":"Scala Jo","address":"in my sweet dream"}]}""")

  val records: List[Map[String, Any]] = for {
    JObject(rec) <- json \ "records"
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

  println(records)

The output of records to screen gives this:

List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))

I want to understand what the lines inside the for loop mean. For example, what is the meaning of this line:

JObject(rec) <- json \ "records"

I understand that the json \ "records" produces a JArray object, but why is it fetched as JObject(rec) at left of <-? What is the meaning of the JObject(rec) syntax? Where does the rec variable come from? Does JObject(rec) mean instantiating a new JObject class from rec input?

BTW, I have a Java programming background, so it would also be helpful if you can show me the Java equivalent code for the loop above.

Vicennial

It produces a JArray of JObjects. So when the <- iterates through it, you can extract the JObjects' content to the rec variable. – Billposter

@GáborBakos: but I don't really get it how could it's possible. Because JArray is not List or iterable object. Also, why I can't do this? for ( rec <- json \ "records", so rec become JObject. What is the reason of JObject(rec) at the left of <- ? – Vicennial

I have to admit not checked the json4s sources yet. The JObject(rec) -I thought- was a result of an unapply method, so rec was declared there. If JArray has a flatMap method, it should just work in for comprehensions. Sorry, I have not enough experience with json4s yet to answer your questions. I hope someone else can give you a proper answer. – Billposter

@GáborBakos: here is the source code, the JArray only have values and apply method. It seems you have knowledge about for (N(x) <- y pattern (assumed N is a class). Can you explain me the pattern? I just want to understand what does the N(x) pattern mean, because I only ever see for (x <- y pattern before. – Vicennial

Here: github.com/json4s/json4s/blob/scala_2.10/ast/src/main/scala/org/… JObject is a case class it automatically get an unapply method, so it can be used in pattern matching/extractors, just like in variable declarations. See: Scala for the Impatient 14.8. – Billposter

@GáborBakos: thanks. What does default unapply method takes and returns if no implementation given? – Vicennial

Let us continue this discussion in chat. – Billposter

Answer 1

You have the following types hierarchy:

  sealed abstract class JValue {
    def \(nameToFind: String): JValue = ???
    def filter(p: (JValue) => Boolean): List[JValue] = ???
  }

  case class JObject(val obj: List[JField]) extends JValue
  case class JField(val name: String, val value: JValue) extends JValue
  case class JString(val s: String) extends JValue
  case class JArray(val arr: List[JValue]) extends JValue {
    override def filter(p: (JValue) => Boolean): List[JValue] = 
      arr.filter(p)
  }

Your JSON parser returns following object:

  object JsonParser {
    def parse(s: String): JValue = {
      new JValue {
        override def \(nameToFind: String): JValue =
          JArray(List(
            JObject(List(
              JField("name", JString("John Derp")),
              JField("address", JString("Jem Street 21")))),
            JObject(List(
              JField("name", JString("Scala Jo")),
              JField("address", JString("in my sweet dream"))))))
      }
    }
  }

  val json = JsonParser.parse("Your JSON")

Under the hood Scala compiler generates the following:

  val res = (json \ "records")
    .filter(_.isInstanceOf[JObject])
    .flatMap { x =>
      x match {
        case JObject(obj) => //
          obj //
            .withFilter(f => f match {
              case JField("name", _) => true
              case _                 => false
            }) //
            .flatMap(n => obj.withFilter(f => f match {
              case JField("address", _) => true
              case _                    => false
            }).map(a => Map(
              "name" -> (n.value match { case JString(name) => name }),
              "address" -> (a.value match { case JString(address) => address }))))
      }
    }

First line JObject(rec) <- json \ "records" is possible because JArray.filter returns List[JValue] (i.e. List[JObject]). Here each value of List[JValue] maps to JObject(rec) with pattern matching.

Rest calls are series of flatMap and map (this is how Scala for comprehensions work) with pattern matching.

I used Scala 2.11.4.

Of course, match expressions above are implemented using series of type checks and casts.

UPDATE:

When you use Json4s library there is an implicit conversion from JValue to org.json4s.MonadicJValue. See package object json4s:

implicit def jvalue2monadic(jv: JValue) = new MonadicJValue(jv)

This conversion is used here: JObject(rec) <- json \ "records". First, json is converted to MonadicJValue, then def \("records") is applied, then def filter is used on the result of def \ which is JValue, then it is again implicitly converted to MonadicJValue, then def filter of MonadicJValue is used. The result of MonadicJValue.filter is List[JValue]. After that steps described above are performed.

Barbee

Wait a sec, in JObject(rec) <- json \ "records", the JArray from json \ "records" is turned into List[JObject] through JArray.filter method. How could that possible? Because when I printed the json \ "records" on RPEL, I got this result: res1: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List(( name,JString(Scala Jo)), (address,JString(in my sweet dream)))))). You see that it returned JArray, not List[JObject]. So how could it turned into List[JObject]? – Vicennial

I think there was no if in suud's example, so withFilter (of filter) would not be the method converting JValue to List[JObject]. – Billposter

Method filter is called on JArray only in for comprehension and its result List[JValue] is used. – Barbee

But if you look at the source code, the JArray / JValue doesn't have filter method. So how could it's possible it can be iterated in for loop? – Vicennial

JArray extends JValue (scala-tools.org/mvnsites/liftweb-2.2/framework/scaladocs/net/…). It contains filter method. It is Lift Json API. – Barbee

But I imported org.json4s.JsonAST._, not lift-json which have different package, and I have checked that I put "org.json4s" %% "json4s-native" % "3.2.11" in sbt, so it's not lift-json that I used. – Vicennial

Also, I just check again the source code, there is no def \ method in the JValue/JArray class too. This is really strange. – Vicennial

>> But if you look at the source code, the JArray / JValue doesn't have filter method. (See my UPDATE). – Barbee

>> Also, I just check again the source code, there is no def \ method in the JValue/JArray class too. (org.json4s.MonadicJValue has def \'. Your json` object is implicitly converted to MonadicJValue' which has def ` and def filter). – Barbee

Ok, I found that in the source code, but what triggers jvalue2monadic implicit method being called? I see jvalue2extractable implicit method also takes JValue input, so what made jvalue2monadic is being called rather than jvalue2extractable? – Vicennial

org.json4s.ExtractableJsonAstNode (result of jvalue2extractable) has no 'def \', by this reason org.json4s.MonadicJValue is used. – Barbee

See my UPDATE with explanations. – Barbee

Hi, I wonder if you can answer this question: is it possible to modify the lines inside the for loop to not using pattern matching? So that means only use simple x <- y pattern instead of JObject(x) <- y. I'm curious because rspencer said it's possible (see why I can't do this? for ( rec <- json \ "records".. part on his answer). – Vicennial

actually your interpretation of scala's compiler output is a bit wrong as JValue has no flatMap actually – Corking

>> actually your interpretation of scala's compiler output is a bit wrong (But List[JValue] has. You can decompile question code and see what happens). – Barbee

Answer 2

You are using a Scala for comprehension and I believe much of the confusion is about how for comprehensions work. This is Scala syntax for accessing the map, flatMap and filter methods of a monad in a concise way for iterating over collections. You will need some understanding of monads and for comprehensions in order to fully comprehend this. The Scala documentation can help, and so will a search for "scala for comprehension". You will also need to understand about extractors in Scala.

You asked about the meaning of this line:

JObject(rec) <- json \ "records"

This is part of the for comprehension.

Your statement:

I understand that the json \ "records" produces a JArray object,

is slightly incorrect. The \ function extracts a List[JSObject] from the parser result, json

but why is it fetched as JObject(rec) at left of <-?

The json \ "records" uses the json4s extractor \ to select the "records" member of the Json data and yield a List[JObject]. The <- can be read as "is taken from" and implies that you are iterating over the list. The elements of the list have type JObject and the construct JObject(rec) applies an extractor to create a value, rec, that holds the content of the JObject (its fields).

how come it's fetched as JObject(rec) at left of <-?

That is the Scala syntax for iterating over a collection. For example, we could also write:

for (x <- 1 to 10)

which would simply give us the values of 1 through 10 in x. In your example, we're using a similar kind of iteration but over the content of a list of JObjects.

What is the meaning of the JObject(rec)?

This is a Scala extractor. If you look in the json4s code you will find that JObject is defined like this:

case class JObject(obj: List[JField]) extends JValue

When we have a case class in Scala there are two methods defined automatically: apply and unapply. The meaning of JObject(rec) then is to invoke the unapply method and produce a value, rec, that corresponds to the value obj in the JObject constructor (apply method). So, rec will have the type List[JField].

Where does the rec variable come from?

It comes from simply using it and is declared as a placeholder for the obj parameter to JObject's apply method.

Does JObject(rec) mean instantiating new JObject class from rec input?

No, it doesn't. It comes about because the JArray resulting from json \ "records" contains only JObject values.

So, to interpret this:

JObject(rec) <- json \ "records"

we could write the following pseudo-code in english:

Find the "records" in the parsed json as a JArray and iterate over them. The elements of the JArray should be of type JObject. Pull the "obj" field of each JObject as a list of JField and assign it to a value named "rec".

Hopefully that makes all this a bit clearer?

it's also helpful if you can show me the Java equivalent code for the loop above.

That could be done, of course, but it is far more work than I'm willing to contribute here. One thing you could do is compile the code with Scala, find the associated .class files, and decompile them as Java. That might be quite instructive for you to learn how much Scala simplifies programming over Java. :)

why I can't do this? for ( rec <- json \ "records", so rec become JObject. What is the reason of JObject(rec) at the left of <- ?

You could! However, you'd then need to get the contents of the JObject. You could write the for comprehension this way:

val records: List[Map[String, Any]] = for {
    obj: JObject <- json \ "records"
    rec = obj.obj
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

It would have the same meaning, but it is longer.

I just want to understand what does the N(x) pattern mean, because I only ever see for (x <- y pattern before.

As explained above, this is an extractor which is simply the use of the unapply method which is automatically created for case classes. A similar thing is done in a case statement in Scala.

UPDATE: The code you provided does not compile for me against 3.2.11 version of json4s-native. This import:

import org.json4s.JsonAST._

is redundant with this import:

import org.json4s._

such that JObject is defined twice. If I remove the JsonAST import then it compiles just fine.

To test this out a little further, I put your code in a scala file like this:

package example

import org.json4s._
// import org.json4s.JsonAST._
import org.json4s.native.JsonParser

class ForComprehension {
  val json = JsonParser.parse(
    """{
      |"records":[
      |{"name":"John Derp","address":"Jem Street 21"},
      |{"name":"Scala Jo","address":"in my sweet dream"}
      |]}""".stripMargin
  )

  val records: List[Map[String, Any]] = for {
    JObject(rec) <- json \ "records"
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

  println(records)
}

and then started a Scala REPL session to investigate:

scala> import example.ForComprehension
import example.ForComprehension

scala> val x = new ForComprehension
List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))
x: example.ForComprehension = example.ForComprehension@5f9cbb71

scala> val obj = x.json \ "records"
obj: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))))

scala> for (a <- obj) yield { a }
res1: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))))

scala> import org.json4s.JsonAST.JObject
for ( JObject(rec) <- obj ) yield { rec }
import org.json4s.JsonAST.JObject

scala> res2: List[List[org.json4s.JsonAST.JField]] = List(List((name,JString(John Derp)), (address,JString(Jem Street 21))), List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))

So:

• You are correct, the result of the \ operator is a JArray
• The "iteration" over the JArray just treats the entire array as the only value in the list
• There must be an implicit conversion from JArray to JObject that permits the extractor to yield the contents of JArray as a List[JField].
• Once everything is a List, the for comprehension proceeds as normal.

Hope that helps with your understanding of this.

For more on pattern matching within assignments, try this blog

UPDATE #2: I dug around a little more to discover the implicit conversion at play here. The culprit is the \ operator. To understand how json \ "records" turns into a monadic iterable thing, you have to look at this code:

• org.json4s package object: This line declares an implicit conversion from JValue to MonadicJValue. So what's a MonadicJValue?
• org.json4s.MonadicJValue: This defines all the things that make JValues iterable in a for comprehension: filter, map, flatMap and also provides the \ and \\ XPath-like operators

So, essentially, the use of the \ operator results in the following sequence of actions: - implicitly convert the json (JValue) into MonadicJValue - Apply the \ operator in MonadicJValue to yield a JArray (the "records") - implicitly convert the JArray into MonadicJValue - Use the MonadicJValue.filter and MonadicJValue.map methods to implement the for comprehension

Sacker

Thanks for the comprehensive answer. But I still don't get it at part of json \ "records", because it returned JArray not List[JObject]. AFAIK, JArray is not a collection, so how could it's possible it can be iterated through <- ? – Vicennial

(suud and me suspected the org.json4s.jvalue2monadic (github.com/json4s/json4s/blob/scala_2.10/core/src/main/scala/…) was in action, but it seems might be other implicit caused it.) – Billposter

I believe that \ just returns a List[JValue] for an array much like \ will return a Map[String,JValue] for an object. I updated my answer to reflect this. – Sacker

But if entered json \ "records" on RPEL, it gave output like this: res1: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List(( name,JString(Scala Jo)), (address,JString(in my sweet dream)))))). So you can see that the json \ "records" is JArray. How could it's converted to List[JObject]? – Vicennial

I investigated with the REPL. See the UPDATE to my answer. – Sacker

Thanks for the update. I still would like to know what made the JArray being converted to List[JObject]. That's the only missing piece now. The other answer said about filter, but I can't found any filter method in JValue class in the source code. – Vicennial

@suud - I added another update to explain the explicit conversions. – Sacker

Thanks for your updated answer :). That's very clear explanation. However I can't mark your answer because srgfed01 had answered first about the MonadicJValue, so I hope you can understand :). BTW, I also want to tell you that your example code about why I can't do this? for ( rec <- json \ "records"... doesn't work when I tried. There is error at this line: rec = obj.rec. The error message said: value rec is not a member of org.json4s.JValue. – Vicennial

The code example has been fixed .. oversight on my part. – Sacker

I got error when I run it: error: type mismatch; found : org.json4s.JsonAST.JObject => (org.json4s.JsonAST.JObject, List[org.json4s.JsonAST.JField]); (which expands to) org.json4s.JsonAST.JObject => (org.json4s.JsonAST.JObject, List[(String, org.json4s.JsonAST.JValue)]); required: org.json4s.JValue => ?; (which expands to) org.json4s.JsonAST.JValue => ?; obj: JObject <- json \ "records" – Vicennial

One of the best written question and answer ever – Sidell

@Sidell - Thank you :) – Sacker

Answer 3

Just simplified example, how for-comprehesion works here:

scala> trait A
defined trait A

scala> case class A2(value: Int) extends A
defined class A2

scala> case class A3(value: Int) extends A
defined class A3

scala> val a = List(1,2,3)
a: List[Int] = List(1, 2, 3)

scala> val a: List[A] = List(A2(1),A3(2),A2(3))
a: List[A] = List(A2(1), A3(2), A2(3))

So here is just:

scala> for(A2(rec) <- a) yield rec //will return and unapply only A2 instances
res34: List[Int] = List(1, 3)

Which is equivalent to:

scala> a.collect{case A2(rec) => rec}
res35: List[Int] = List(1, 3)

Collect is based on filter - so it's enough to have filter method as JValue has.

P.S. There is no foreach in JValue - so this won't work for(rec <- json \ "records") rec. But there is map, so that will: for(rec <- json \ "records") yield rec

If you need your for without pattern matching:

for {
   rec <- (json \ "records").filter(_.isInstanceOf[JObject]).map(_.asInstanceOf[JObject])
   rcobj = rec.obj 
   name <- rcobj if name._1 == "name" 
   address <- rcobj if address._1 == "address" 
   nm = name._2.asInstanceOf[JString].s
   vl = address._2.asInstanceOf[JString].s
} yield Map("name" -> nm, "address" -> vl) 

res27: List[scala.collection.immutable.Map[String,String]] = List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))

Corking

Thanks, your example is very simple and clear :). But about foreach in JValue, when I tried for(rec <- json \ "records") yield rec, I got result in screen: res29: org.json4s.JValue = JArray(List(JObject(List((id,JString(001)), (desc,JString(test1)))), JObject(List((id,JString(002)),(desc,JString(in test2)))))). Maybe you were mistaking with for(rec <- json \ "records") println(rec)? – Vicennial

Oh. My bad - I've written yield in my answer but didn't mean it actually :) – Corking

Ok, no problem :). Now is it possible for you to create same output with for { x <- y, ..} yield Map(..) pattern? That means no pattern matching like JObject(rec) <- .. inside the for loop. That's the only thing left that made me curious as rspencer said it's possible. – Vicennial

Without any pattern matching: for(rec <- (json \ "records").filter(_.isInstanceOf[JObject]).map(_.asInstanceOf[JObject]); name <- rec.obj.find(_._1 == "name"); addr <- rec.obj.find(_._1 == "address")) yield name -> addr. – Corking

That's close, the expected output is: List[Map[String,Any]] = List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream)). What you returned is still List of Map[String,JField]. Btw, why there must be semicolon between lines in the for comprehension? – Vicennial

for(rec <- (json \ "records").filter(_.isInstanceOf[JObject]).map(_.asInstanceOf[JObject]); rcobj = rec.obj; name <- rcobj.find(_._1 == "name"); addr <- rcobj.find(_._1 == "address"); nm = name._2.asInstanceOf[JString].s; vl = addr._2.asInstanceOf[JString].s) yield Map("name" -> nm, "address" -> vl) res27: List[scala.collection.immutable.Map[String,String]] = List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream)) I've used semicolon just to write it as one-liner – Corking

Great, you made it :). Could you put the code in your answer so anyone can read it better? Strange, if I don't put the semicolon, I will get compiler error: ')' expected but '<-' found.. Do you know why? Could it be your codes inside for is not for-comprehension? – Vicennial

Ah, I see. You are using "(" and ")" so that's why it needs semicolon. If I use "{" and "}", it doesn't need it. – Vicennial

Updated. Just to finish it with something about Category theory (feeling like Sheldon with his "Fun With Flags" podcast) - There is no foreach in MonadicJValue type class because foreach assumes side-effects, but there is no side-effects in pure-FP. But this is not actually Monadic as there is no flatMap method or at least some concat between JsValues (as it could give you flatten in combination with fold) – Corking

Many thanks :). Is it possible on this line: rec <- (json \ "records").filter(_.isInstanceOf[JObject]).map(_.asInstanceOf[JObject]) can be made shorter maybe by using if ..., is it possible? About Monadic, is it has same meaning with Monad? Is Monad in Scala an object thas has map, flatMap, and filter operation? – Vicennial

same, flatMap is enough for monad. If you have map only it's just a Functor. – Corking

Wait, at this line (name, nm) <- rcobj, for the (name, nm) tuple, isn't it kinda like a pattern matching? In this case, is it a JField(x,y) matching? Also, for the if name == "name" you don't wrap it with find method at this time, is it because find and filter are same? (I heard if .. in for-comprehension is translated as filter) – Vicennial

if you have only one matched element - find and filter will be logically same. removed tuples – Corking

So the tuple is something related with the category theory you mentioned before? Could you explain a bit about the category theory? – Vicennial

tuples - it's kind of products in category theory. You can also check this question/answer about analogies #27752924 – Corking

Should I read a basic scala book first (maybe like martin odersky's book) before reading Learning Scalaz? With z letter behind (like Dragon Ball Z) sounds hardcore to me :) – Vicennial

You should :). But you could also combine scalaz with wikipedia and read it randomly. – Corking

Scalaz - is actually Haskell's library adopted to the Scala – Corking

Is category theory something like shortcut? So here we have X (JObject) -> Y (JField) -> Z (JString). The shortcut path is X -> Z which produces X -> (Y, Z). – Vicennial

Category theory is something like theory - so you should spent a lot of time to understand what it is :). Anyway it's all about object's and arrows in most abstract meaning, so there is no specifications about shortcuts or not. If you mean commutative diagrams - their only restriction is commutation - composition of different paths (with any length) in the diagram should give the same result. In other words any possible way (composition of morphisms) from A to B should give you the B. – Corking