Can Map be performed on a Scala HList

Question

I have done a few implementations of HList now. One based on Daniel Spiewak's High Wizardry in the Land of Scala talk and another based on a post in Apocalisp blog. The goal was to have a heterogenous list of which is not heterogenous in the primary type but rather the higher kind. For example:

val requests = Request[String] :: Request[Int] :: HNil

I would be able to do a map across the list to perform the request and result in a heterogenous list of the higher kind. So:

requests.map(execute)

should equal

String :: Int :: HNil

Sadly all my attempts have resulted in an HList of Any. Here is the code from a recent attempt:

class Request[+Out](o:Out) {
  type O = Out

  def v:O = o
}

object HList {
  trait Func[-Elem,Out] {
    type Apply[E <: Elem] <: Out
    def apply[N <: Elem](e:N):Apply[N]
  }
  sealed trait HList[Base] {
    type Head <: Base
    type Tail <: HList[Base]
    type Map[Out,F <: Func[Base,Out]] <: HList[Out]
    def head:Head
    def tail:Tail

    def ::[A <: Base](a:A):HList[Base]
    def map[Out,F <: Func[Base,Out]](f:F):Map[Out,F]
  }

  case class HNil[Base]() extends HList[Base] {
    type Head = Nothing
    type Tail = Nothing
    type Map[Out,F <: Func[Base,Out]] = HNil[Out]

    def head = error("Head of an empty HList")
    def tail = error("Head of an empty HList")

    def ::[A <: Base](a:A) = HCons(a,this)
    def map[Out,F <: Func[Base,Out]](f:F) = new HNil[Out]
  }

  case class HCons[Base,A <: Base,B <: HList[Base]](head: A, tail: B) extends HList[Base] {
    type Head = A
    type Tail = B    
    type Map[Out,F <: Func[Base,Out]] = HCons[Out,F#Apply[Head],Tail#Map[Out,F]]

    def ::[C <: Base](c:C) = HCons(c,this)
    def map[Out,F <: Func[Base,Out]](f:F) =
      HCons(f(head),tail.map(f))
  }

  val :: = HCons 
}

object Test extends Application {
  import HList._

  val HNil = new HNil[Request[_]]

  val list = new Request[Int](1) :: new Request[String]("1") :: HNil

  val (a :: b :: HNil) = list
  val y:Request[String] = b

  val results = list.map[Any,Unwrap.type](Unwrap)

  val i:Int = results.head
}

import HList._
object Unwrap extends Func[Request[Any],Any] {
  type Apply[I <: Request[Any]] = I#O
  def apply[N <: Request[Any]](e:N) = null.asInstanceOf[Apply[N]]
}

The other attempt was based on the Apocalisp version which uses fold to create a new HList and again it resulted in a HList of Any types. Any tips would be appreciated.

Lobation

Answer 1

The HList implementation in shapeless is rich enough to subsume both HList and KList functionality. It provides a map operation which applies a higher-ranked function, possibly with type-specific cases, across it's elements yielding an appropriately typed HList result,

import shapeless.Poly._
import shapeless.HList._

// Define a higher-ranked function from Sets to Options
object choose extends (Set ~> Option) {
  def default[T](s : Set[T]) = s.headOption 
}

// An HList of Sets
val sets = Set(1) :: Set("foo") :: HNil

// Map our choose function across it ...
val opts = sets map choose

// The resulting value
opts == Option(1) :: Option("foo") :: HNil 

Note that although it's the case in the above example there's no requirement that the HList elements share a common outer type constructor, it just has to be the case that the higher-ranked function mapped with has cases for all of the types involved,

// size is a higher-ranked function from values of arbitrary type to a 'size'
// which is defined as 1 by default but which has type specific cases for
// Strings and tuples
object size extends (Id ~> Const[Int]#λ) {
  def default[T](t : T) = 1
}
implicit def sizeString = size.λ[String](s => s.length)
implicit def sizeTuple[T, U](implicit st : size.λ[T], su : size.λ[U]) =
  size.λ[(T, U)](t => 1+size(t._1)+size(t._2))

size(23) == 1          // Default
size("foo") == 3       // Type specific case for Strings
size((23, "foo")) == 5 // Type specific case for tuples

Now let's map this across an HList,

val l = 23 :: true :: "foo" :: ("bar", "wibble") :: HNil
val ls = l map size

ls == 1 :: 1 :: 3 :: 10 :: HNil

In this case the result type of the function being mapped is constant: it's an Int no matter what the argument type is. Consequently the resulting HList has elements all of the same type, which means that it can usefully be converted to a vanilla list,

ls.toList == List(1, 1, 3, 10)

Browne

Is the definition of size valid in Shapeless 2.0? I'm getting an error "object creation impossible, since method apply in trait ~> of type [T](f: shapeless.Id[T])Int is not defined" – Cory

In case anyone is wondering, the error @JamesMoore reported is solved by replacing default with apply – Berthaberthe

Anyone know what the current syntax for this is? – Taintless

Looks like the current syntax for this required defining a case for each type you can be applied to? See books.underscore.io/shapeless-guide/… – Dietsche

Actually I was able to get the current syntax for the higher-ranked function working via import poly._. I think the ~> operator may have been moved into the poly package? – Dietsche

Answer 2

what you need is a Klist with type constructor Request, and a natural transformation execute: Request ~> Id. All of this is detailed in the marvelous type-level programming series of posts at Apocalisp, in particular:

1. Natural transformation literals
2. Klist basics

you can checkout the code for the whole series from Mark Harrah's up repo

In your case, you'll need something like

val reqList = new Request[Int](1) :^: new Request[String]("1") :^: KNil    
val exec = new (Request ~> Id) { def apply[T](reqs: Request[T]): T = reqs.execute }    
val results = reqList down exec

the down method above is conceptually the same as map for a nat transf M ~> Id; you also have more general map which from a nat transf M ~> N and a Klist of kind M yields a KList of kind N.

Prostyle

I had noticed KLists in XSBT but haven't had time to take a look. Thanks for the tips. This issue was driving me craze – Lobation

Answer 3

Note that you have an example of Map with HList in the recent (October 2016, 5 years after the OP) article "Using shapeless' HLists for extra type safety (in Akka Streams)" from Mikołaj Koziarkiewicz.

  //glue for the ParserStageDefs
  specs.map(s => Flow[Data].map(s.parser).map(s.processor))
                    .foreach(broadcast ~> _ ~> merge)

The problem lies in the fact that the type information in our specs list is not preserved. Or rather, not preserved the way we want to - the type of the List elements is ParserStageDef[_ >: Int with String], so the lowest common supertype for our decorator and incrementer.

The above implies that, when mapping between the parser and processor elements, the compiler has no way to provide the actual type T that's used within the given spec.

A solution

Here's where HLists come to the rescue. Because they preserve the complete type information for each element, it's possible to define our flow very similarly to our last attempt.

First, let's replace our list with an HList:

import shapeless.ops.hlist._
import shapeless._
//...

val specs = decorator :: incrementer :: HNil
val specsSize = specs.length.toInt

Now, for the mapping from ParserStageDefs into Flows, we need to take a different approach, as the map for HList requires something called P**oly - a polymorphic function value**.

Here's how one would look like in our case:

import shapeless.PolyDefns.~>
object toFlow extends (ParserStageDef ~> ProcessingFlow) {
  override def apply[T](f: ParserStageDef[T]) = 
                Flow[Data].map(f.parser).map(f.processor)
}

For it to work, we'll also have change ProcessingFlow to type ProcessingFlow[_] = Flow[Data, Data, _], since the polymorphic function above expects a higher-kinded type.

Now, our central statement turns out to be:

//we convert to a List[ProcessingFlow[_]] for simplicity
specs.map(toFlow).toList.foreach(broadcast ~> _ ~> merge)

and we're all set!

Pedroza