How to log all headers in nginx?

Question

How do I log all the headers the client (browser) has sent in Nginx? I also want to log the response headers. Note that I am using nginx as reverse proxy.

After going through documentation, I understand that I can log a specific header, but I want to log all of the headers.

Paulinapauline

Valid http headers can be very long and write them to nginx access.log is a bad idea. If you need this for debugging something, use tcpdump with filters. – Tejeda

@Tejeda I understand. But I am bound by certain limitations and really need to log all HTTP headers in nginx for proof of concept project. If it is not possible with nginx, I will indeed go with other options. – Paulinapauline

Generaly is not possible, but you can write you module :) – Tejeda

Answer 1

After much research, I can conclude that it is not possible out of the box.

Update- you can use openresty which comes with Lua. Using Lua one can do pretty cool things, including logging all of the headers to say, Redis or some other server

Paulinapauline

there are answers below that work out of the box with stock nginx - see @Tozzart's answer using the error log in debug mode. – Faze

Answer 2

There are two more options to log headers in nginx.

• nginx njs scripting language can be used instead of lua (njs may be considered easier to install and to use and is somewhat more "official")
• mirror directive (only for request headers)

Using njs to log request and response headers

njs can be installed from package repositories and is preinstalled in official nginx docker images. njs is available since at least 1.9.15 version of nginx (which is rather old), but it is better to use more recent versions.

After installation enable njs http module in nginx config:

load_module modules/ngx_http_js_module.so;

Headers may be logged to error or access log.

Logging to access log with njs

• Decide which format to use (JSON, custom, base64...)
• Create js module with function to convert headers structure to string (~3 lines of code)
• Import this js module in nginx configuration (~1 line)
• Declare a variable to use in log_format directive (~1 line)
• Add this variable to log format (~1 line)

HTTP Request object has headersIn, headersOut fields in key value format, duplicate headers are merged in this fields and rawHeadersIn, rawHeadersOut which are array of arrays of raw headers.

Create js module, use json to serialize headers:

// /etc/nginx/headers.js
function headers_json(r) {
  return JSON.stringify(r.headersIn)
}

export default {headers_json};

Import js module, declare variable and add it to log_format:

http {
  js_import headers.js;
  js_set $headers_json headers.headers_json;

  # Using custom log format here
  log_format  main  '$remote_addr'
                    '\t$remote_user'
                    '\t$time_local'
                    '\t$request'
                    '\t$status'
                    '\t$headers_json';

Escaping in acccess log

By default strings in access log are escaped, so you will get something like this:

 # curl -H 'H: First' -H 'H: Second' localhost:8899
 172.17.0.1      -       16/Apr/2021:08:46:43 +0000      GET / HTTP/1.1  200     {\x22Host\x22:\x22localhost:8899\x22,\x22User-Agent\x22:\x22curl/7.64.1\x22,\x22Accept\x22:\x22*/*\x22,\x22H\x22:\x22First,Second\x22}

You can use escape parameter in log_format directive to change how escaping is applied.

Example of escape=json output:

log_format main escape=json ...
{\"Host\":\"localhost:8899\",\"User-Agent\":\"curl/7.64.1\",\"Accept\":\"*/*\",\"H\":\"First,Second\"}

Another option is to wrap json in base64 encoding in javascript function:

function headers_json_base64(r) {
  return JSON.stringify(r.headersIn).toString('base64')
}

Logging to error log with njs

With njs you can use ngx.log or r.log (for older versions of njs ngx object is not available) in javascript function to log headers. js function should be called explicitly for this to work for example with js_header_filter directive.

js module:

function headers_json_log(r) {
    return ngx.log(ngx.WARN, JSON.stringify(r.headersIn))
}
export default {headers_json_log};

Enable logging in location:

location /log/ {
  js_header_filter headers.headers_json_log;
  return 200;
}

For error log escaping is not applied, so you will get raw json:

2021/04/16 12:22:53 [warn] 24#24: *1 js: {"Host":"localhost:8899","User-Agent":"curl/7.64.1","Accept":"*/*","H":"First,Second"}

Logging to error log may be useful if you don't want to mess up your access logs or you need to log headers only for specific location (for specific location access_log directive and separate log_format can be used too)

Logging request headers with mirror directive

mirror directive can be used to mirror requests to different location. It may be more convenient than tcpdump especially when upstream traffic is encrypted and is a bit simpler than using njs.

mirror can be used only to capture request headers since response headers are returned independently.

mirror directive can be used server wide in http and server contexts or in location context.

# using mirror in server context
mirror /mirror;
mirror_request_body off;

location /mirror {
    # Use internal so that location is not available for direct requests
    internal;
    # Use small timeout not to wait for replies (this is not necessary)
    proxy_read_timeout 1;
    # Pass headers to logging server
    proxy_pass http://127.0.0.1:6677;
    # send original request uri in special header
    proxy_set_header X-Original-URI $request_uri;
}

To log headers simple http server or just netcat oneliner may be used:

nc -kl 6677 > /path/to/headers.log

Because netcat doesn't reply to nginx, nginx will fill up error log with timeout errors, this errors do not affect clients.

Greyhound

Excellent! Mirroring to nc is super easy to set up and it told me exactly what I wanted to know. – Womera

Nice ! Although '# /etc/nginx/headers.js' throws an error in a JS file. The '#' is not a comment or Shebang ? Pretty easy to install. I take it that Lua is part of NGINX plus and not the standard NGINX ? You need OpenResty for that ? – Desensitize

Fixed # comment – Greyhound

Lua is not part of nginx or nginx plus it is completely separate project based on nginx called OpenResty. njs (nginx javascript) is part of nginx since pretty long time (not so long as openresty exists, but also many years) and as such is available in almost all nginx distributions by default (docker or as a package) – Greyhound

One tip, I didn't make it work until restarted nginx. – Alrich

Answer 3

As @gauravphoenix said you need openresty which comes with Lua. See https://github.com/openresty/lua-nginx-module/ for installing it. Once it's running then add in nginx

header_filter_by_lua_block {
  local h = ngx.req.get_headers()
  for k, v in pairs(h) do
    ngx.log(ngx.ERR, "Got header "..k..": "..tostring(v)..";")
  end
}

Inspect your error log.

Granivorous

a more useful variation of this would be to inject them into an nginx var you can consume in your log_format custom - otherwise you'd have one hell of a time trying to relate a random header log to a particular request, even on a slow server.. :) (if I get such a thing setup, I'll share it) – Pleomorphism

I want to add to this answer, that some values can be of a table type, which can raise an exception. To overcome this it's better to use tostring( v ) instead of just v. – Smaragdine

This code (at time of comment) uses toString(v) which my version of openresty doesn't have. My version of openresty uses tostring(v), which makes the code above throw lots of errors and makes me sad. – Transfigure

Answer 4

For request headers only it's possible out of the box on nginx/1.18.0.

1. Edit /etc/nginx/nginx.conf line error_log /var/log/nginx/error.log;. Move it outside the http/server scope so it looks like:

error_log /var/log/nginx/error.log debug;

http {
  ...

Notice the debug part!

2. Restart the server and then do tail -f /var/log/nginx/error.log | grep "http header"

Output example:

2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "Host: example.org"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "Connection: Keep-Alive"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "X-Forwarded-For: 127.0.0.1"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "X-Forwarded-Proto: https"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "user-agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
...

Tumid

Thanks for this! For anyone reading this who is running nginx from a docker container the instructions in the 'Running nginx in debug mode' section of (hub.docker.com/_/nginx) will help. Essentially you need to run the 'nginx-debug' binary rather than the 'nginx' binary for debug mode to work – Liva

Thanks for this, this really helped me solve the issue! – Carcajou

Answer 5

Based on @user1778602’s response the set_by_lua_block can be used to set all headers into a variable to be consumed later at the log_format (see more details in this answer).

set_by_lua_block $request_headers{
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    local rowtext = ""
    rowtext = string.format("[%s %s]\n", k, v)
    request_headers_all = request_headers_all .. rowtext

  end
  return request_headers_all
}

The update your log_format (be sure to use the new format in e.g. access_log):

log_format myformat escape=none "$request_headers"

PS: Beware of logging PII data may be violating GDPR :)

Cacoepy

Thanks a lot, this helped me out. – Alongside

As a side comment, my header did not appear because it contained an underscore. I had to enable this directive to make it work: enable-underscores-in-headers github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/… – Alongside

And additional side note: special characters \n will fail when using set_by_lua. See my solution (if this answer update is rejected) : https://mcmap.net/q/243316/-lua-concatenation-of-variables-pairs-escaping-special-characters-openresty-nginx . The important thing is to use set_by_lua_block instead and then make sure log_format escape=none switch – Repartition

Answer 6

I don't know if it's a new feature since the OP asked the question, but you can simply turn on debug level logging, and NGINX will log all request headers (plus a whooole lot more) to the error log.

error_log /var/log/nginx/error.log debug;
#                                  ^^^^^

(then restart nginx)

I wouldn't leave this enabled, since (1) it may log sensitive data and (2) it will fill up your disk very quickly.

Abduct

This works fine for request headers, but OP stated "I also want to log the response headers". Unfortunately I'm in the same boat. If only it was this easy. – Understate

@RinoBino it is. Response headers are definitely logged. In the debug log, they'll appear pretty much verbatim as how the browser sees them. E.g. you'll see a "HTTP/1.1 200 OK" (or whatever the response is) followed by the headers sent to the browser. – Abduct

Answer 7

If you really want to log all client and server headers without so much trouble (like using $upstream_http_server), see: https://github.com/openresty/lua-nginx-module#header_filter_by_lua

Here's an implementation example:

        set $req_headers "";
        set $resp_headers "";
        header_filter_by_lua_block {
          local cjson = require "cjson"

          local h = ngx.req.get_headers()
          ngx.var.req_headers = cjson.encode(h)

          local rh = ngx.resp.get_headers()
          ngx.var.resp_headers = cjson.encode(rh)
        }

then include the Lua variables in a pretty json log format:

        log_format all_headers_log_format escape=json
              '{'
                '"timestamp":"$msec",'
                '"remote_addr":"$remote_addr",'
                '"remote_user":"$remote_user",'
                '"host":"$host",'
                '"status":"$status",'
                '"req_headers":"$req_headers",'
                '"resp_headers":"$resp_headers"'
              '}';
        access_log  /var/log/nginx/access.log  all_headers_log_format;

Tourane