How does one log Akka HTTP client requests

Question

I need to log akka http client requests as well as their responses. While there seems to be a hint of API for logging these requests, there is no clear documentation on how it should be done. My approach has been to create a logged request which transparently wraps Http().singleRequest(req) as follows:

def loggedRequest(req: HttpRequest)
                  (implicit system: ActorSystem, ctx: ExecutionContext, m: Materializer): Future[HttpResponse] = {

  Http().singleRequest(req).map { resp ⇒
    Unmarshal(resp.entity).to[String].foreach{s ⇒
      system.log.info(req.toString)
      system.log.info(resp.toString + "\n" + s)
    }
    resp
  }
}

Unfortunately, I have to grab the future either through the unmarshal or by simply requesting resp.entity.dataBytes in order to recover the body of the response. I get the logging but the promise gets completed and I can no longer unmarshal the entity to the actual data. A working solution would log the request and response and pass this test case without an IllegalStateException with "Promise already completed" being thrown:

describe("Logged rest requests") {

  it("deliver typed responses") {
    val foo = Rest.loggedRequest(Get(s"http://127.0.0.1:9000/some/path"))
    val resp = foo.futureValue(patience)
    resp.status shouldBe StatusCodes.OK
    val res = Unmarshal(resp.entity).to[MyClass].futureValue
  }
}

Ideas welcome.

Endways

I'm trying to do the same. Did you find a solution? – Prana

Answer 1

One of the solution I've found is to use a:

import akka.http.scaladsl.server.directives.DebuggingDirectives

val clientRouteLogged = DebuggingDirectives.logRequestResult("Client ReST", Logging.InfoLevel)(clientRoute)
Http().bindAndHandle(clientRouteLogged, interface, port)

Which can easily log the request and result in raw(bytes) format. The problem is that those logs are completely unreadable. And here is place where it became complicated.

Here is my example that encode the entity of the request/response and write it to the logger.

You can pass a function to:

DebuggingDirectives.logRequestResult

def logRequestResult(magnet: LoggingMagnet[HttpRequest ⇒ RouteResult ⇒ Unit])

That is function written using magnet pattern:

LoggingMagnet[HttpRequest ⇒ RouteResult ⇒ Unit]

Where:

LoggingMagnet[T](f: LoggingAdapter ⇒ T)

Thanks to that we have access to all parts that we need to log the request and result. We have LoggingAdapter, HttpRequest and RouteResult

In my case I've create an inside function. I don't want to pass all the parameters again.

def logRequestResult(level: LogLevel, route: Route)
                      (implicit m: Materializer, ex: ExecutionContext) = {
  def myLoggingFunction(logger: LoggingAdapter)(req: HttpRequest)(res: Any): Unit = {
    val entry = res match {
      case Complete(resp) =>
        entityAsString(resp.entity).map(data ⇒ LogEntry(s"${req.method} ${req.uri}: ${resp.status} \n entity: $data", level))
      case other =>
        Future.successful(LogEntry(s"$other", level))
    }
    entry.map(_.logTo(logger))
  }
  DebuggingDirectives.logRequestResult(LoggingMagnet(log => myLoggingFunction(log)))(route)
}

The most important part is the last line where I put myLoggingFunction in to logRequestResult.

The function called myLoggingFunction, simple matched the result of server computation and create a LogEntry based on it.

The last thing is a method that allows to decode the result entity from a stream.

def entityAsString(entity: HttpEntity)
                   (implicit m: Materializer, ex: ExecutionContext): Future[String] = {
entity.dataBytes
  .map(_.decodeString(entity.contentType().charset().value))
  .runWith(Sink.head)
}

The method can be easily add to any akka-http route.

val myLoggedRoute = logRequestResult(Logging.InfoLevel, clientRoute)
Http().bindAndHandle(myLoggedRoute, interface, port)

Spermogonium

Please correct typos. Please correct indention. And maybe add a little more information on what exactly your code is doing. – Childers

AFAICS the question was about logging requests and responses at the client, while this answer is about logging requests and responses at the server, right? – Keening

This is too much work just for cross cutting concern like logging. It should be simple. Besides, I agree with Arnout that this doesn't provide solution about logging the client requests. – Dolmen

@ArnoutEngelen AFAICS the question didn't ask for "logging at the client" but "logging client requests" where "client" is redundant. so "how to log requests and responses" would be more accurate – Sines

@Sines indeed the question text is somewhat ambiguous, but the code describing the posters' approach so far shows the Akka HTTP client API. – Keening

The question is about HTTP Client requests, and code samples used are for HTTP Client. I was looking for the same and this answer is wrong. – Arminius

This answer is incorrect as the question is about sending requests, not receiving – Suisse

The wisdom of the crowd... in the comments, it is. – Checkoff

Answer 2

For another solution, this code logs the request IP and associates a random number with each request and response so they can be associated in the logs. It also records the response time.

Since the request may take awhile to process, and may fail, I wanted to see the request immediately, and see the response if and when it returns.

RequestFields is just the data I care about from the request. There's a lot of noise by default.

val logRequestResponse: Directive0 =
  extractRequestContext flatMap { ctx =>
    extractClientIP flatMap { ip =>
      val id = scala.math.abs(rand.nextLong).toString
      onSuccess(RequestFields.fromIdIpAndRequest(id, ip, ctx.request)) flatMap { req =>
        logger.info("request", req.asJson)
        val i = Instant.now()
        mapRouteResultWith { result => 
          Result.fromIdStartTimeAndRouteResult(id, i, result) map { res =>
            logger.info("response", res.asJson)
            result
        }
      }
    }
  }
}

Forgotten

Thanks sean - would it be possible to post the complete code, including imports? It's unclear what is custom code on your part and what is build-in akka code. – Visa

Answer 3

My complete solution, inspired by @seanmcl

trait TraceDirectives extends LazyLogging {

  private val counter: AtomicLong = new AtomicLong(0)

  def log: Directive0 = count flatMap { requestId =>
    mapInnerRoute(addLoggingToRoute(requestId, _))
  }

  private def count: Directive1[Long] = Directive { innerRouteSupplier =>
    ctx =>
      innerRouteSupplier(Tuple1(counter.incrementAndGet()))(ctx)
  }

  private def addLoggingToRoute(requestId: Long, innerRoute: Route): Route = {
    ctx => {
      val requestStopwatch = Stopwatch.createStarted()
      extractClientIP { ip =>
        logger.info("Http request, id: {}, uri: {}, forwarded ip: {}", requestId, ctx.request.uri, ip)
        mapResponse(httpResponse => {
          logger.info("Http response, id: {}, code: {}, time: {}", requestId, httpResponse.status.intValue(), requestStopwatch.toString)
          httpResponse
        })(innerRoute)
      }(ctx)
    }
  }
}

object TraceDirectives extends TraceDirectives

Crossopterygian

How can I implement i in my route ? – Aesthesia

I've fixed log to public. Use it as any other directive: post { IctTraceDirectives.log { complete(null) } } – Crossopterygian

You should show the imports: import java.util.concurrent.atomic.AtomicLong import akka.http.scaladsl.server.{Directive, Directive0, Directive1, Route} import akka.http.scaladsl.server.Directives._ import com.google.common.base.Stopwatch import com.typesafe.scalalogging.LazyLogging ... and mention the dependency: "com.typesafe.scala-logging" %% "scala-logging" % "3.9.2" – Timaru

Answer 4

I was able to get wire-level logging from Akka HTTP client by setting the config value akka.http.client.log-unencrypted-network-bytes to a integer value (chunk size in bytes) as decsribed here.

P.S. I know this is a very old question, but it's where I landed after searching for "akka http client wire logging", so I'm contributing the solution that I found for others that may land here also.

Fructuous

Thanks for this update. Do you have an example of how you would be able to extract a Request from the log so as to be able to log specific details? – Endways