How to convert JSON to scala shapeless.hlist?

Question

I got json like {"name":"susan","age":25},and a hint to json keyset like "name:String,age:Int",how to create a HList from that json?

Tracitracie

Can you use a library? or it has to be done by hand? if you can use a library have yu already selected that library or can we suggest one? Also, are you sure an HList is your end goal or maybe a case class? – Canella

want to parse json dynamically to hlist – Tracitracie

Shapeless is compile safe, JSON isn't. Just to be clear, there is nothing "automatic" you can do (I.e. You can't make something that will magically guess the types of a JSON String) – Ardys

but i got a hint of json value type,it's a text like "name:String,age:Int",how to use that to specify type?@Ardys – Tracitracie

Here there is a brief of Dave Gurnell´s shapeless book(chapter 5): https://mcmap.net/q/67541/-using-shapeless-hlist-to-easily-build-json-decoder – Gabo

@LuisMiguelMejíaSuárez I want to convert json values to Tuple or case class,don't know how to do that – Tracitracie

I would recommend you to take a look to any json library like circe. – Canella

That sounds like another problem. It may be better to expand this question with all details, or maybe close this one and open a new one to pick more attention. Anyways, I do not know Flink so I can not help anymore, sorry. – Canella

circe.github.io/circe – Jehial

@LuisMiguelMejíaSuárez do u know how to convert json to HList with that hint "name:String,age:Int"? – Tracitracie

Yeah a library like circe reads a json a converts it into an HList then it can convert that into a case class, not sure how to use that with Flink or even if that is recommended. – Canella

thank you@LuisMiguelMejíaSuárez – Tracitracie

@LuisMiguelMejíaSuárez "library like circe reads a json a converts it into an HList then it can convert that into a case class" A type of hlist (String :: Int :: HNil) or case class (A for case class A(s: String, i: Int)) has to be known at compile time. But hint "name:String,age:Int" seems to be a runtime String. – Jehial

@DmytroMitin yeah, I really did not understand what exactly OP wanted to do. So that is why I suggested opening a new question with more details. – Canella

@DmytroMitin I guess he wants to parse a string to case class. If it is not in the correct format - he has to deal with that as with error (Either.left, Option.none or an exception). – Pibroch

@ArtemSokolov Yeah, I guess I understood, thanks, see the answer. – Jehial

@Tracitracie Actually, I guess we do someting strange. We use highly type-level libraries aimed to static type safety and then run them at runtime with reflection ignoring type safety and getting actually List[Any] (HList). See update of my answer. – Jehial

@DmytroMitin thank you so much,but my end goal is tuple or case class,can you see my other question?linke – Tracitracie

@Tracitracie If you goal is a tuple or case class why is it written "How to convert JSON to scala shapeless.hlist?", "how to create a HList from that json?" in your question? It's not difficult to modify my answer in order to have a tuple or case class rather than HList. But again, similarly to having HList and not having String :: Int :: HNil you'll have (Any, Any) and not have (String, Int). – Jehial

@DmytroMitin Thanks again,I'm new with scala reflect,how to modify you answer to have a tuple or case class? – Tracitracie

@Tracitracie See update. – Jehial

Answer 1

Based on the code you added and then deleted, it seems, having a runtime-string hint "name:String,age:Int" and runtime-string json {"name":"susan","age":25}, you want to get an HList with runtime reflection. You can do this as follows

import shapeless.HList
import scala.reflect.runtime
import scala.reflect.runtime.universe._
import scala.tools.reflect.ToolBox
val tb = runtime.currentMirror.mkToolBox()

val jsonStr = """{"name":"susan","age":25}"""
val hint = "name:String,age:Int"
val classType = tb.define(tb.parse(s"case class Test($hint)").asInstanceOf[ImplDef]).asClass.toType
val hlist = tb.eval(q"""
  import io.circe.generic.auto._
  import io.circe.parser.decode
  val classInstance = decode[$classType]($jsonStr)

  import shapeless.Generic
  Generic[$classType].to(classInstance.toOption.get)
""").asInstanceOf[HList]
println(hlist) // susan :: 25 :: HNil

Please notice that you do everything at runtime so you'll have no access to type String :: Int :: HNil at compile time and hlist has static type just HList (not String :: Int :: HNil) and HList is actually not better than just List[Any].

build.sbt

libraryDependencies ++= Seq(
  scalaOrganization.value % "scala-reflect"  % scalaVersion.value,
  scalaOrganization.value % "scala-compiler" % scalaVersion.value,
  "com.chuusai" %% "shapeless" % "2.4.0-M1",
  "io.circe" %% "circe-core"    % "0.13.0",
  "io.circe" %% "circe-parser"  % "0.13.0",
  "io.circe" %% "circe-generic" % "0.13.0"
)

---

Actually, I guess we do someting strange. We use highly type-level libraries (shapeless, circe) aimed to static type safety and then run them at runtime with reflection ignoring type safety and getting actually List[Any] (HList).

I guess that if List[Any] (list of field values) is enough for you then you just need to use a more runtime library. For example, with json4s

import org.json4s.{JInt, JObject, JString, JValue}
import org.json4s.jackson.JsonMethods._

val jsonStr: String = """{"name":"susan","age":25}"""
val json: JValue = parse(jsonStr) //JObject(List((name,JString(susan)), (age,JInt(25))))
val l: List[JValue] = json.asInstanceOf[JObject].obj.map(_._2) //List(JString(susan), JInt(25))
val res: List[Any] = l.map {
  case JString(s) => s
  case JInt(n)    => n
} //List(susan, 25)

build.sbt

libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.6.9"

---

Actually the same can be done with Circe, just with parse instead of decode[A]

import io.circe.{Json, JsonNumber}
import io.circe.parser.parse

val jsonStr: String = """{"name":"susan","age":25}"""
val json: Json = parse(jsonStr).toOption.get //{"name":"susan","age":25}
val l: List[Json] = json.asObject.get.values.toList //List("susan", 25)
val res: List[Any] = l.map(_.fold[Any](null, null, (_: JsonNumber).toInt.get, identity[String], null, null)) //List(susan, 25)

---

If you need an instance of case class or a tuple rather than HList replace

tb.eval(q"""
  import io.circe.generic.auto._
  import io.circe.parser.decode
  val classInstance = decode[$classType]($jsonStr)

  import shapeless.Generic
  Generic[$classType].to(classInstance.toOption.get)
""").asInstanceOf[HList] // susan :: 25 :: HNil

with

tb.eval(q"""
  import io.circe.generic.auto._
  import io.circe.parser.decode
  decode[$classType]($json).toOption.get
""").asInstanceOf[Product] //Test(susan,25)

or

tb.eval(q"""
  import io.circe.generic.auto._
  import io.circe.parser.decode
  val classInstance = decode[$classType]($json)

  import shapeless.Generic
  Generic[$classType].to(classInstance.toOption.get).tupled
""").asInstanceOf[Product] //(susan,25)

correspondingly.

Jehial