How to get around the Scala case class limit of 22 fields?

Question

Scala case classes have a limit of 22 fields in the constructor. I want to exceed this limit, is there a way to do it with inheritance or composition that works with case classes?

Brimmer

Yes your right. In some cases I use an implicit case class which uses less fields than the actual class. – Brimmer

Answer 1

More recently (Oct 2016, six years after the OP), the blog post "Scala and 22" from Richard Dallaway explores that limit:

Back in 2014, when Scala 2.11 was released, an important limitation was removed:

Case classes with > 22 parameters are now allowed. 

That said, there still exists a limit on the number of case class fields, please see https://mcmap.net/q/411589/-what-is-the-maximum-number-of-case-class-fields-in-scala

This may lead you to think there are no 22 limits in Scala, but that’s not the case. The limit lives on in functions and tuples.

The fix (PR 2305) introduced in Scala 2.11 removed the limitation for the above common scenarios: constructing case classes, field access (including copying), and pattern matching (baring edge cases).

It did this by omitting unapply and tupled for case classes above 22 fields.
In other words, the limit to Function22 and Tuple22 still exists.

Working around the Limit (post Scala 2.11)

There are two common tricks for getting around this limit.

• The first is to use nested tuples.
  Although it’s true a tuple can’t contain more than 22 elements, each element itself could be a tuple

• The other common trick is to use heterogeneous lists (HLists), where there’s no 22 limit.

If you want to make use of case classes, you may be better off using the shapeless HList implementation. We’ve created the Slickless library to make that easier. In particular the recent mappedWith method converts between shapeless HLists and case classes. It looks like this:

import slick.driver.H2Driver.api._
import shapeless._
import slickless._

class LargeTable(tag: Tag) extends Table[Large](tag, "large") {
  def a = column[Int]("a")
  def b = column[Int]("b")
  def c = column[Int]("c")
  /* etc */
  def u = column[Int]("u")
  def v = column[Int]("v")
  def w = column[Int]("w")

  def * = (a :: b :: c :: /* etc */ :: u :: v :: w :: HNil)
    .mappedWith(Generic[Large])
}

There’s a full example with 26 columns in the Slickless code base.

Locksmith

Answer 2

This issue is going to be fixed in Scala 2.11.

Pepperandsalt

scala 2.11 is life saver! – Hoad

Answer 3

Build a normal class that acts like a case class.

I still use scala 2.10.X since that is what is the latest supported by Spark, and in Spark-SQL I make heavy use of case classes.

The workaround for case classes with more than 22 fields:

class Demo(val field1: String,
    val field2: Int,
    // .. and so on ..
    val field23: String)

extends Product 
//For Spark it has to be Serializable
with Serializable {
    def canEqual(that: Any) = that.isInstanceOf[Demo]

    def productArity = 23 // number of columns

    def productElement(idx: Int) = idx match {
        case 0 => field1
        case 1 => field2
        // .. and so on ..
        case 22 => field23
    }
}

Infer

Extending Product gives you the nice iteration parts, but you don't get the copy method. The copy method is in case classes but not the Product trait, because it is generated by the Scala compiler (that's now it can be strongly typed for each field in your case class). – Crosseyed

What serializer do you register with Spark? I have started with twitter.chill.avro.AvroSerializer.SpecificRecordBinarySerializer, but in that case, my class must implement SpecificRecordBase, which puts me up against the 22 field limit again. – Jeanelle

Answer 4

It's interesting your constructor is that loaded, but you could package related values into a case class of their own.

So while you might have

case class MyClass(street: String, city: String, state: String, zip: Integer)

you can do this

case class MyClass(address: Address)

You have other options too:

• Group items into tuples
• Create your own Function23 trait (or whatever)
• Use currying

UPDATE: As others have noted, this is no longer an issue after the release of Scala 2.11--though I would hesitate to use the term "fix." However, the "Catch 22," if you will, sometimes still shows up in third-party Scala libraries.

Tacheometer

Answer 5

Try doing something like this if it matches your use case.

val schema: StructType = StructType(
    Array(
      StructField(name = "name", StringType),
      StructField(name = "size", StringType)
    )
 )

https://mcmap.net/q/188833/-encoders-product-of-a-scala-trait-schema-in-spark

Bootblack

Answer 6

Good to know:

In Scala 3, Case class can now have any number of parameters. Functions beyond scala.Function22 are erased to a new trait scala.runtime.FunctionXXL.

Cockatrice

Answer 7

When you have that many values, it's usually a sign that your design needs to be reworked anyways.

Form intermittent case classes that then aggregate into the larger one. This also makes the code much easier to understand, reason about, and maintain. As well as bypassing this issue you are having.

For example, if I wanted to store user data I might do this....

case class User(name: Name, email: String)
case class Name(first: String, last: String)

With so few things, this of course wouldn't be necessary. But if you have 22 things you are trying to cram into one class, you'll want to do this sort of intermittent case class-work anyways.

Broomrape

There is a single json that needs to be parsed which has more than 22 fields. Using an object mapper like json4s it requires a case a class with all fields. – Substitution

this answer was the first thought that came to me and it is more or less correct – Mcdougall