How to make nginx print full log for tcp stream

Question

I'm using nginx-1.11.8 with the following configuration.

stream {

    log_format  basic   '$time_iso8601 $remote_addr '
                        '$protocol $status $bytes_sent $bytes_received '
                        '$session_time $upstream_addr '
                        '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log      logs/stream.log  basic buffer=1k flush=5s;

    include *.stream.conf;
}

Now I can only get IP and other unimportant stuff in the tcp log. There are some important information related to the IP in the tcp data packet that I want to know. What should I do to get the full tcp packet in the tcp log? Thanks in advance.

Brasier

It's surprising how hard it is to find this info for stream access_log. All the search results just give http or error_log configuration. – Courtesan

did you find solution to this? expecially logging the request. – Eagan

Do you want the packets (including headers) or just the payload? Nginx is not a packet sniffer, it is using sockets and won't necessarily have access to the other parts of the packets (except socket identifiers, like IP addresses and ports). If you need just the payload, do you actually need the individual packet payloads or just the reconstructed data stream? As a socket client, nginx won't know the packet boundaries for TCP. – Carolynncarolynne

@Carolynncarolynne Sorry it's been so long...I thought what I wanted was the individual packet payloads, to be specific, the payment request data. The TCP module we were using was not actually for a real "stream", it was just used to send payment requests, and the interval between two requests can be seconds, so it's not necessary for the Nginx to know the packet boundaries for TCP. – Brasier

@Brasier Given that you only need the stream(ish) payload, I was hoping there would be something in openresty's ngx_stream_lua_module but at least at present I see nothing directly helpful. The closest is preread_by_lua_block where you can hijack the socket, but unfortunately once hijacked there doesn't seem to be a way to hand control, or at least the data you peeked at, back for proxy_pass and later directives. And that's with openresty; there's definitely nothing in stock nginx. – Carolynncarolynne

Answer 1

I don't think this is possible with stock nginx, although it may someday be possible with openresty. The ngx_stream_lua_module is still in its infancy and may yet develop features to support peeking at stream content. They might be accepting feature requests, too.

As to why e.g. $request_body doesn't exist in stream context today, I think there are a few reasons:

• Stream support is still somewhat new, compared to HTTP support
• Nginx doesn't speak the underlying protocol (of course, you could put HTTP traffic through a stream block but nginx won't handle it any differently than non-HTTP traffic)
• As a consequence, there are questions nginx can't categorically answer, like:
  1. What is a request?
  2. Where does it start?
  3. Where does it end?
  4. Does it even have something like a "body"?
  5. Is there even such a thing as a "request"?
• In http context, nginx already needed to implement buffering of requests to support retrying them with multiple server lines in an upstream block when one fails. It's not safe to do this with arbitrary TCP protocols (heck, it's not even safe to do this with HTTP in all circumstances, but that's the server admin's concern). No such feature exists presently in stream context, so there's no buffer to piggy back off of for a variable.

For some elaboration on those questions nginx can't answer, which may seem silly at first, consider that TCP is a pretty generic protocol. The simplest application layer protocols built on top of TCP define a very basic request-response model where one end sends a plain text request followed by a delimiter, while the other side waits for the request then sends a plain text response followed by a delimiter, then one or both sides close the connection. In those cases, sure, it seems straightforward to capture and log the request and response.

However, lots of more complicated protocols don't work that way. Some protocols don't even operate on a request-reply model at all and instead are asynchronous, where either side can send a "message" at any time, like WebSockets, STOMP, or AMQP. There may be no delimiters in the stream at all, or the delimiters could be hidden under layers of compression and encryption, and not all network encryption is SSL/TLS. Moreover, what if the content is encoded as ASN.1, Protobuf, Thrift, Avro, or something like that? You'd probably want a nice pretty printed form but nginx can't do that for you, and dumping a bunch of binary data in the log will be difficult if not impossible to reliably parse. There could also be nested "channels" or "sessions" like with SSH. Heck, even with request-response models, which side makes the requests and which side sends the responses could get flipped over the lifetime of the connection.

All that having been said, I think there is an argument for handling the common or at least simple cases, so maybe this will make it into nginx some day. Until then, it's probably easiest to put an intermediary either in front of or behind nginx using socat or another network debugging tool.

Carolynncarolynne

Answer 2

As in the comment, NGINX is not by design a packet sniffer.

You can have a look at supported variables you can display in the logs:

http://nginx.org/en/docs/varindex.html

Bousquet

In particular, $request_body exists in http context but unfortunately is not defined for stream context logging – Carolynncarolynne