nginx monitoring response from upstream server

Question

I have a reverse proxy setup with nginx.

Client ------> Nginx ------------------------------------> Backend Server
       <------       <-----------------------------------
                      (I want to see the requests here)

How can I log the http requests including headers sent from the backend server to nginx into a file?

Maybe one of the directives in nginx http proxy module can help me do this.

But I cannot find any helpful directives.

Chandlery

Answer 1

Note: I gave this answer before the OP added tag openresty. I am not an expert in Openresty, but I am sure I gave correct details in case of vanilla Nginx.

---

A "backend server" is called "upstream" in Nginx terminology. And there is a separate page in the documentation that lists variables associated w/ upstreams. Among them, you will probably be interested in

• $upstream_addr – IP address of an backend server that processed the request
• $upstream_connect_time, $upstream_header_time, $upstream_response_time – how long Nginx waited for connection accept, for header from the upstream and how long did it take for the backend to process the request
• $upstream_http_*NAME* will contain the response header *NAME*. For example, $upstream_http_etag or $upstream_http_last_modified.
• there are other variables to report caching status, retries, etc.

It's quite common practice to log these into Nginx log file. You will need to declare your own log format, for example:

log_format my_upstream '$remote_addr [$time_local] "$request" $status'
  '"$upstream_addr" $upstream_response_time $upstream_http_etag';

and then use it in location or server:

access_log /var/log/nginx/upstream.log my_upstream;

Goodfornothing

It's strange that my upstream_* value always displays -. e.g. upstream_addr, upstream_http_host, and upstream_connect_time – Chandlery

It's definitely not about log file location. Some variables may contain - when they make no sense. E.g. upstream_response_time makes no sense in case of serving static files or when Nginx returned a response from its cache. – Goodfornothing

I think my problem is access_log get triggered before nginx receives the response from the upstream. That's why my upstream_addr has no value. – Chandlery

I did not see your configuration, so I have no idea what's happening on your side. What I know for sure is that (a) Nginx is one of the most stable pieces of software I've met; (b) these variables work; (c) it's possible to screw everything, including most stable pieces of software. – Goodfornothing

Is there a way we can access the headers sent from nginx to an upstream server? I have checked the code and upstream struct has only headers_in and no headers_out. – K

Answer 2

If you can accept OpenResty:

Within https://github.com/openresty/lua-nginx-module#body_filter_by_lua_block you may reconstruct a full body if you want to include it to the log. When eof flag is specified you may get all response's header (https://github.com/openresty/lua-nginx-module#ngxrespget_headers)

Ten set a variable in Lua and use it within log_format.

Attention - your access log file may be huge!

Hint - you may use separate access_log and log_format for response logging.

Sippet

This only applies to response headers back to the client (not between proxy and upstream). – Cortes