Lua - Concatenation of variables (pairs) escaping special characters - Openresty Nginx

Question

I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.

Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.

This answer shows how to concatenate headers into a string $request_headers:

set_by_lua $request_headers '
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
  end
  return request_headers_all
';

I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.

Log format:

log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**

Host: "192.168.1.4", Via : "-", 
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one

When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.

I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?

What is the "..variablename.." doing, is this how variables are always used inside Lua strings?

How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?

Thompkins

github.com/openresty/… TL;DR: use set_by_lua_block {} – Interlining

I wish you answered earlier ;) ...only required to use double escaping sigh! I came up with the same conclusion that I needed set_by_lua_block after investigating further. I used string.format, but if I realised double escaping can work would have been easier! I never saw your message until now after I replied with my solution. For anyone else see my answer below : 1. set_by_lua_block and 2. log_format escape=none – Thompkins

Answer 1

The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).

Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.

I will attempt to update the title to reflect more specific requirements

TLDR

0. Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
1. Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
2. Update nginx configuration with your custom/existing log_format and add the switch escape=none.

Full Explanation

Investigating the provided link answer set_by_lua function documentation :

NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.

So from the original set_by_lua from the link:

set_by_lua $request_headers '
  return 'stringvalue'
';

I changed to set_by_lua_block function:

this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)

set_by_lua_block $request_headers{
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    local rowtext = ""
    rowtext = string.format("[%s %s]\n", k, v)
    request_headers_all = request_headers_all .. rowtext

  end
  return request_headers_all
}

The important part is that this _block {} function escapes the special characters correctly.

After that I received output in the log files as : x0A (newline character literal).

The final step then is to update the nginx.conf file with the custom log_format and add escape=none:

log_format log_realip escape=none "$nginx_variables"

Thompkins

Answer 2

request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"

contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"

Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.

What is the "..variablename.." doing, is this how variables are always used inside Lua strings?

using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.

Read https://www.lua.org/manual/5.4/manual.html#3.4.6

Burrows

As with the other answer above: I am 100% sure I tried that format as well. Just to make sure I copied your format : failed to load inlined Lua code: set_by_lua:5: unfinished string near '"]', client: 192.168.1.3, server: 192.168.1.4, request: "GET /phpinfo.php HTTP/1.1", host: "192.168.1.4". In essence I agree with your observation that `\n"]" should not work because it's 'part of the variable output'... but it doesn't crash the page. Your solution gives the error I just displayed. This behaviour is why I asked the question in the first place. – Thompkins

Your explanation helps me understand the various different behaviour depending on the type. I will investigate on how to edit the string to make openresty understand the string correctly. – Thompkins

Answer 3

you add the \n to a wrong place, you can change to request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.

In lua, the .. is a concat operator, to concat to strings, for example:

print("hello" .. "world")

get the result helloworld.

your code \n"]" have syntax error, because \n not in a string.

lua strings can not directly use variables, usually, lua use string.format for complex string. for example:

local test = "hello"
string.format("%s world",test) -- hello world

you can use string.format for you string concat. also, you can use table.concat to concat strings. for example:

local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world

Choosey

I am 100% sure I tried that format as well. Just to make sure I copied your format : failed to load inlined Lua code: set_by_lua:5: unfinished string near '"]', client: 192.168.1.3, server: 192.168.1.4, request: "GET /phpinfo.php HTTP/1.1", host: "192.168.1.4". In essence I agree with your observation that `\n"]" should not work because it's 'part of the variable output'... but it doesn't crash the page. Your solution gives the error I just displayed. This behaviour is why I asked the question in the first place. – Thompkins

I will investigate how to close the unfinished string further. – Thompkins

I will try string.format or table.concat to see if any of them will work better in Openresty blocks – Thompkins

The unfinished string error was easy to found if you copy you lua code to a editor with syntax prompt like vscode. your error unfinished string near '"]' seems lack of the last " – Choosey

Answer 4

The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?

Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.

Edit: I missed that this was already answered by using the other parsing function.

Additionally the docs state to use the original parsing function escapes need to be double escaped.

Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.

Agriculture

Thank you Kirin ;) As already explained, I have tried those formats as well - "\n" outside the array, which makes sense to all of us, (I am aware my original example shows it inside!). Thanks for the explanation.... but : The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?. This is where the issue is: Openresty+lua (wrapper software) probably handles it strangely. I found the solution with the other function in Openresty. But yes the core issue - based on my original example ... – Thompkins

.... everyone (including even me in the future) would assume I made a mistake on the original question with the format. I tried several different types, none worked. Had to use the other Openresty function. – Thompkins

Ohh my bad, I see it now, to do it using your original implementation you double escape it so it would be \\\\n according to the docs. " – Agriculture