Writing a test case for file uploads in Play 2.1 and Scala

Question

I found the following question/answer:

Test MultipartFormData in Play 2.0 FakeRequest

But it seems things have changed in Play 2.1. I've tried adapting the example like so:

"Application" should {

"Upload Photo" in {
  running(FakeApplication()) {
    val data = new MultipartFormData(Map(), List(
        FilePart("qqfile", "message", Some("Content-Type: multipart/form-data"), 
            TemporaryFile(getClass().getResource("/test/photos/DSC03024.JPG").getFile()))
        ), List())
    val Some(result) = routeAndCall(FakeRequest(POST, "/admin/photo/upload", FakeHeaders(), data)) 
    status(result) must equalTo(CREATED)
    headers(result) must contain(LOCATION)
    contentType(result) must beSome("application/json")  

However whenever I attempt to run the request, I get a null-pointer exception:

[error] ! Upload Photo
[error]     NullPointerException: null (PhotoManagementSpec.scala:25)
[error] test.PhotoManagementSpec$$anonfun$1$$anonfun$apply$3$$anonfun$apply$4.apply(PhotoManagementSpec.scala:28)
[error] test.PhotoManagementSpec$$anonfun$1$$anonfun$apply$3$$anonfun$apply$4.apply(PhotoManagementSpec.scala:25)
[error] play.api.test.Helpers$.running(Helpers.scala:40)
[error] test.PhotoManagementSpec$$anonfun$1$$anonfun$apply$3.apply(PhotoManagementSpec.scala:25)
[error] test.PhotoManagementSpec$$anonfun$1$$anonfun$apply$3.apply(PhotoManagementSpec.scala:25)

If I try to replace the deprecated routeAndCall with just route (and remove the Option around result), I get a compile error stating that it can't write an instance of MultipartFormData[TemporaryFile] to the HTTP response.

What's the right way to design this test in Play 2.1 with Scala?

---

Edit: Tried to modify the code to test just the controller:

"Application" should {

"Upload Photo" in {

   val data = new MultipartFormData(Map(), List(
   FilePart("qqfile", "message", Some("Content-Type: multipart/form-data"), 
    TemporaryFile(getClass().getResource("/test/photos/DSC03024.JPG").getFile()))
), List())

   val result = controllers.Photo.upload()(FakeRequest(POST, "/admin/photo/upload",FakeHeaders(),data))


   status(result) must equalTo(OK)
   contentType(result) must beSome("text/html")
   charset(result) must beSome("utf-8")
   contentAsString(result) must contain("Hello Bob")
  }

But I now get a type error on all the test conditions around the results like so:

[error]  found   : play.api.libs.iteratee.Iteratee[Array[Byte],play.api.mvc.Result]
[error]  required: play.api.mvc.Result

I don't understand why I'm getting an Interator for byte arrays mapped to Results. Could this have something to do with how I'm using a custom body parser? My controller's definition looks like this:

def upload = Action(CustomParsers.multipartFormDataAsBytes) { request =>

  request.body.file("qqfile").map { upload =>

Using the form parser from this post: Pulling files from MultipartFormData in memory in Play2 / Scala

Diaconate

A similar question was asked and answered: #15013677 – Chorion

I see the question and answer, but it's still really confusing and not answered well. You point to the official documentation, which I've read, and which doesn't cover multi-part form data. I would actually like to test the route as well, but I guess testing the controller will do. I still do not understand how to pass the file data to the body with the name "qqfile." Could you edit your question with a full answer? – Diaconate

Attempted to test just the controller, but still ran into some issues. Edit is listed above. – Diaconate

I have removed the first part of that answer (which was wrong). – Chorion

The error you are now getting is caused by the way your upload method method is called. You call it with a TemporaryFile, but you specified a multipartFormDataAsBytes body parser. You should call it with an Array[Byte] instead of a TemporaryFile as data. – Chorion

This question shouldn't be a duplicate. It's nothing like the question it says it's a duplicate of. This question is asking specifically about test cases. The other question is a general question about form uploads. – Diaconate

Answer 1

Play 2.3 includes a newer version of httpmime.jar, requiring some minor corrections. Building on Marcus's solution using Play's Writeable mechanism, while retaining some of the syntactic sugar from my Play 2.1 solution, this is what I've come up with:

import scala.language.implicitConversions

import java.io.{ByteArrayOutputStream, File}

import org.apache.http.entity.ContentType
import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.entity.mime.content._
import org.specs2.mutable.Specification

import play.api.http._
import play.api.libs.Files.TemporaryFile
import play.api.mvc.MultipartFormData.FilePart
import play.api.mvc.{Codec, MultipartFormData}
import play.api.test.Helpers._
import play.api.test.{FakeApplication, FakeRequest}

trait FakeMultipartUpload {
  implicit def writeableOf_multiPartFormData(implicit codec: Codec): Writeable[MultipartFormData[TemporaryFile]] = {
    val builder = MultipartEntityBuilder.create().setBoundary("12345678")

    def transform(multipart: MultipartFormData[TemporaryFile]): Array[Byte] = {
      multipart.dataParts.foreach { part =>
        part._2.foreach { p2 =>
          builder.addPart(part._1, new StringBody(p2, ContentType.create("text/plain", "UTF-8")))
        }
      }
      multipart.files.foreach { file =>
        val part = new FileBody(file.ref.file, ContentType.create(file.contentType.getOrElse("application/octet-stream")), file.filename)
        builder.addPart(file.key, part)
      }

      val outputStream = new ByteArrayOutputStream
      builder.build.writeTo(outputStream)
      outputStream.toByteArray
    }

    new Writeable[MultipartFormData[TemporaryFile]](transform, Some(builder.build.getContentType.getValue))
  }

  /** shortcut for generating a MultipartFormData with one file part which more fields can be added to */
  def fileUpload(key: String, file: File, contentType: String): MultipartFormData[TemporaryFile] = {
    MultipartFormData(
      dataParts = Map(),
      files = Seq(FilePart[TemporaryFile](key, file.getName, Some(contentType), TemporaryFile(file))),
      badParts = Seq(),
      missingFileParts = Seq())
  }

  /** shortcut for a request body containing a single file attachment */
  case class WrappedFakeRequest[A](fr: FakeRequest[A]) {
    def withFileUpload(key: String, file: File, contentType: String) = {
      fr.withBody(fileUpload(key, file, contentType))
    }
  }
  implicit def toWrappedFakeRequest[A](fr: FakeRequest[A]) = WrappedFakeRequest(fr)
}

class MyTest extends Specification with FakeMultipartUpload {
  "uploading" should {
    "be easier than this" in {
      running(FakeApplication()) {
        val uploadFile = new File("/tmp/file.txt")
        val req = FakeRequest(POST, "/upload/path").
          withFileUpload("image", uploadFile, "image/gif")
        val response = route(req).get
        status(response) must equalTo(OK)
      }
    }
  }
}

Supremacist

Thanks, works a treat on 2.3 and saved me much bother. – Allocution

Can confirm that this works on 2.4 either. I wonder why there is no Writeable for MultipartFormData in Helpers... – Judkins

Answer 2

I managed to get this working with Play 2.1 based on various mailing list suggestions. Here's how I do it:

import scala.language.implicitConversions

import java.io.{ ByteArrayOutputStream, File }

import org.apache.http.entity.mime.MultipartEntity
import org.apache.http.entity.mime.content.{ ContentBody, FileBody }
import org.specs2.mutable.Specification

import play.api.http.Writeable
import play.api.test.{ FakeApplication, FakeRequest }
import play.api.test.Helpers._

trait FakeMultipartUpload {
  case class WrappedFakeRequest[A](fr: FakeRequest[A]) {
    def withMultipart(parts: (String, ContentBody)*) = {
      // create a multipart form
      val entity = new MultipartEntity()
      parts.foreach { part =>
        entity.addPart(part._1, part._2)
      }

      // serialize the form
      val outputStream = new ByteArrayOutputStream
      entity.writeTo(outputStream)
      val bytes = outputStream.toByteArray

      // inject the form into our request
      val headerContentType = entity.getContentType.getValue
      fr.withBody(bytes).withHeaders(CONTENT_TYPE -> headerContentType)
    }

    def withFileUpload(fileParam: String, file: File, contentType: String) = {
      withMultipart(fileParam -> new FileBody(file, contentType))
    }
  }

  implicit def toWrappedFakeRequest[A](fr: FakeRequest[A]) = WrappedFakeRequest(fr)

  // override Play's equivalent Writeable so that the content-type header from the FakeRequest is used instead of application/octet-stream  
  implicit val wBytes: Writeable[Array[Byte]] = Writeable(identity, None)
}

class MyTest extends Specification with FakeMultipartUpload {
  "uploading" should {
    "be easier than this" in {
      running(FakeApplication()) {
        val uploadFile = new File("/tmp/file.txt")
        val req = FakeRequest(POST, "/upload/path").
          withFileUpload("image", uploadFile, "image/gif")
        val response = route(req).get
        status(response) must equalTo(OK)
      }
    }
  }
}

Supremacist

I'll give this a try. I've posted a solution I got to work as well. – Diaconate

Cannot make it work. Getting exception IllegalArgumentException: protocol = http host = null. Currently using "commons-httpclient" % "commons-httpclient" % "3.1". Is this the culprit or is it something else? – Yul

Answer 3

I've modified Alex's code to act as a Writable which better integrates into Play 2.2.2

package test

import play.api.http._
import play.api.mvc.MultipartFormData.FilePart
import play.api.libs.iteratee._
import play.api.libs.Files.TemporaryFile
import play.api.mvc.{Codec, MultipartFormData }
import java.io.{FileInputStream, ByteArrayOutputStream}
import org.apache.commons.io.IOUtils
import org.apache.http.entity.mime.MultipartEntity
import org.apache.http.entity.mime.content._

object MultipartWriteable {

  /**
   * `Writeable` for multipart/form-data.
   *
   */
  implicit def writeableOf_multiPartFormData(implicit codec: Codec): Writeable[MultipartFormData[TemporaryFile]] = {

    val entity = new MultipartEntity()

    def transform(multipart: MultipartFormData[TemporaryFile]):Array[Byte] = {

      multipart.dataParts.foreach { part =>
        part._2.foreach { p2 =>
            entity.addPart(part._1, new StringBody(p2))
        }
      }

      multipart.files.foreach { file =>
        val part = new FileBody(file.ref.file, file.filename,     file.contentType.getOrElse("application/octet-stream"), null)
        entity.addPart(file.key, part)
      }

      val outputStream = new ByteArrayOutputStream
      entity.writeTo(outputStream)
      val bytes = outputStream.toByteArray
      outputStream.close
      bytes
    }

    new Writeable[MultipartFormData[TemporaryFile]](transform, Some(entity.getContentType.getValue))
  }
}

This way it is possible to write something like this:

val filePart:MultipartFormData.FilePart[TemporaryFile] = MultipartFormData.FilePart(...)
val fileParts:Seq[MultipartFormData.FilePart[TemporaryFile]] = Seq(filePart)
val dataParts:Map[String, Seq[String]] = ...
val multipart = new MultipartFormData[TemporaryFile](dataParts, fileParts, List(), List())
val request = FakeRequest(POST, "/url", FakeHeaders(), multipart)

var result = route(request).get

Thimbleweed

If you add something like this to the above you can actually use Play's FakeRequest().withMultipartFormDataBody() method. It's bizarre that it's broken out of the box. – Hid

Answer 4

Following EEColor's suggestion, I got the following to work:

"Upload Photo" in {


    val file = scala.io.Source.fromFile(getClass().getResource("/photos/DSC03024.JPG").getFile())(scala.io.Codec.ISO8859).map(_.toByte).toArray

    val data = new MultipartFormData(Map(), List(
    FilePart("qqfile", "DSC03024.JPG", Some("image/jpeg"),
        file)
    ), List())

    val result = controllers.Photo.upload()(FakeRequest(POST, "/admin/photos/upload",FakeHeaders(),data))

    status(result) must equalTo(CREATED)
    headers(result) must haveKeys(LOCATION)
    contentType(result) must beSome("application/json")      


  }

Diaconate

It's worth pointing out that this solution bypasses Play's router and serialization while the one from Alex does not. – Bono

For me, this one still doesn't work while Alex's version works and looks cleaner to me. You can re-use it. Your version is complaining about not having a Array[Byte] Writer... – Tripos

Answer 5

Here's my version of Writeable[AnyContentAsMultipartFormData]:

import java.io.File

import play.api.http.{HeaderNames, Writeable}
import play.api.libs.Files.TemporaryFile
import play.api.mvc.MultipartFormData.FilePart
import play.api.mvc.{AnyContentAsMultipartFormData, Codec, MultipartFormData}

object MultipartFormDataWritable {
  val boundary = "--------ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890"

  def formatDataParts(data: Map[String, Seq[String]]) = {
    val dataParts = data.flatMap { case (key, values) =>
      values.map { value =>
        val name = s""""$key""""
        s"--$boundary\r\n${HeaderNames.CONTENT_DISPOSITION}: form-data; name=$name\r\n\r\n$value\r\n"
      }
    }.mkString("")
    Codec.utf_8.encode(dataParts)
  }

  def filePartHeader(file: FilePart[TemporaryFile]) = {
    val name = s""""${file.key}""""
    val filename = s""""${file.filename}""""
    val contentType = file.contentType.map { ct =>
      s"${HeaderNames.CONTENT_TYPE}: $ct\r\n"
    }.getOrElse("")
    Codec.utf_8.encode(s"--$boundary\r\n${HeaderNames.CONTENT_DISPOSITION}: form-data; name=$name; filename=$filename\r\n$contentType\r\n")
  }

  val singleton = Writeable[MultipartFormData[TemporaryFile]](
    transform = { form: MultipartFormData[TemporaryFile] =>
      formatDataParts(form.dataParts) ++
        form.files.flatMap { file =>
          val fileBytes = Files.readAllBytes(Paths.get(file.ref.file.getAbsolutePath))
          filePartHeader(file) ++ fileBytes ++ Codec.utf_8.encode("\r\n")
        } ++
        Codec.utf_8.encode(s"--$boundary--")
    },
    contentType = Some(s"multipart/form-data; boundary=$boundary")
  )
}

implicit val anyContentAsMultipartFormWritable: Writeable[AnyContentAsMultipartFormData] = {
  MultipartFormDataWritable.singleton.map(_.mdf)
}

It's adapted from (and some bugs fixed): https://github.com/jroper/playframework/blob/multpart-form-data-writeable/framework/src/play/src/main/scala/play/api/http/Writeable.scala#L108

See the whole post here, if you are interested: http://tech.fongmun.com/post/125479939452/test-multipartformdata-in-play

Skydive

Answer 6

For me, the best solution for this problem is the Alex Varju one

Here is a version updated for Play 2.5:

object FakeMultipartUpload {
  implicit def writeableOf_multiPartFormData(implicit codec: Codec): Writeable[AnyContentAsMultipartFormData] = {
    val builder = MultipartEntityBuilder.create().setBoundary("12345678")

    def transform(multipart: AnyContentAsMultipartFormData): ByteString = {
      multipart.mdf.dataParts.foreach { part =>
        part._2.foreach { p2 =>
          builder.addPart(part._1, new StringBody(p2, ContentType.create("text/plain", "UTF-8")))
        }
      }
      multipart.mdf.files.foreach { file =>
        val part = new FileBody(file.ref.file, ContentType.create(file.contentType.getOrElse("application/octet-stream")), file.filename)
        builder.addPart(file.key, part)
      }

      val outputStream = new ByteArrayOutputStream
      builder.build.writeTo(outputStream)
      ByteString(outputStream.toByteArray)
    }

    new Writeable(transform, Some(builder.build.getContentType.getValue))
  }
}

Harrold

Answer 7

In Play 2.6.x you can write test cases in the following way to test file upload API:

class HDFSControllerTest extends Specification {
  "HDFSController" should {
    "return 200 Status for file Upload" in new WithApplication {

      val tempFile = SingletonTemporaryFileCreator.create("txt","csv")
      tempFile.deleteOnExit()

      val data = new MultipartFormData[TemporaryFile](Map(),
      List(FilePart("metadata", "text1.csv", Some("text/plain"), tempFile)), List())

      val res: Option[Future[Result]] = route(app, FakeRequest(POST, "/api/hdfs").withMultipartFormDataBody(data))
      print(contentAsString(res.get))
      res must beSome.which(status(_) == OK)
   }
  }
}

Puett

Answer 8

Made Alex's version compatible with Play 2.8

import akka.util.ByteString
import java.io.ByteArrayOutputStream
import org.apache.http.entity.mime.content.StringBody
import org.apache.http.entity.ContentType
import org.apache.http.entity.mime.content.FileBody
import org.apache.http.entity.mime.MultipartEntityBuilder
import play.api.http.Writeable
import play.api.libs.Files.TemporaryFile
import play.api.mvc.Codec
import play.api.mvc.MultipartFormData
import play.api.mvc.MultipartFormData.FilePart
import play.api.test.FakeRequest

trait FakeMultipartUpload {

  implicit def writeableOf_multiPartFormData(
    implicit codec: Codec
  ): Writeable[MultipartFormData[TemporaryFile]] = {
    val builder = MultipartEntityBuilder.create().setBoundary("12345678")

    def transform(multipart: MultipartFormData[TemporaryFile]): ByteString = {
      multipart.dataParts.foreach { part =>
        part._2.foreach { p2 =>
          builder.addPart(part._1, new StringBody(p2, ContentType.create("text/plain", "UTF-8")))
        }
      }
      multipart.files.foreach { file =>
        val part = new FileBody(
          file.ref.file,
          ContentType.create(file.contentType.getOrElse("application/octet-stream")),
          file.filename
        )
        builder.addPart(file.key, part)
      }

      val outputStream = new ByteArrayOutputStream
      builder.build.writeTo(outputStream)
      ByteString(outputStream.toByteArray)
    }

    new Writeable(transform, Some(builder.build.getContentType.getValue))
  }

  /** shortcut for generating a MultipartFormData with one file part which more fields can be added to */
  def fileUpload(
    key: String,
    file: TemporaryFile,
    contentType: String
  ): MultipartFormData[TemporaryFile] = {
    MultipartFormData(
      dataParts = Map(),
      files = Seq(FilePart[TemporaryFile](key, file.file.getName, Some(contentType), file)),
      badParts = Seq()
    )
  }

  /** shortcut for a request body containing a single file attachment */
  case class WrappedFakeRequest[A](fr: FakeRequest[A]) {
    def withFileUpload(key: String, file: TemporaryFile, contentType: String) = {
      fr.withBody(fileUpload(key, file, contentType))
    }
  }
  implicit def toWrappedFakeRequest[A](fr: FakeRequest[A]) = WrappedFakeRequest(fr)
}

Selfdelusion