Use newline with jq

Question

I've seen a number of posts on this but can't figure out what I need exactly. I've tried -r and argjson among other things.

I need the newlines to remain as \n and not be escaped to \\n.

I'd also like to be able to use ``` for code blocks but it ignores that section of the string.

FALLBACK_MESSAGE="TEST MESSAGE - $HOSTNAME"
MARKDOWN_MESSAGE="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <@U12345789>\n```Can we do a\nmultiline code block```"
JSON_STRING=$( jq -nr \
    --arg fallbackMessage "$FALLBACK_MESSAGE" \
    --arg sectionType "section" \
    --arg markdownType "mrkdwn" \
    --arg textMessage "$MARKDOWN_MESSAGE" \
    '{
        text: $fallbackMessage, 
        blocks: [
            {
                type: $sectionType,
                text: {
                    type: $markdownType, 
                    text: $textMessage
                }
            }
        ]
    }')
echo $JSON_STRING

Outputs:

{ "text": "TEST MESSAGE - devDebug", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "TEST MESSAGE - devDebug \\(0x0a) \\(\\n) Hi <@U12345789>\\n" } } ] }

Yah

Put a newline in your string as a newline literal, not the two characters backslash and n. – Heller

That is to say, newline=$'\n', or printf -v newline '%b' '\n' will make a shell variable newline with an actual newline in it. – Heller

BTW, don't use all-caps names -- they're reserved for variables built into (or meaningful to) the shell itself. – Heller

...remember, you're telling jq to convert literal data into JSON. The JSON form of the string \n is "\\n", whereas the JSON form of a string with only a newline is "\n". So you want the data you're putting into jq to have the literal newline that's ready to be converted, not the two-character \n sequence that isn't. – Heller

Also, in double quotes, backticks are treated as command substitution syntax, so you need to escape them to stop their contents from being run as code. – Heller

Note for zsh users: I'm trying to do the same but zsh behaviors are inconsistent: #73899610 – Parma

Answer 1

Make sure your shell variables contain actual newlines, not \n sequences.

If you want bash to convert escape sequences in a string into the characters they refer to, printf %b can be used for this purpose.

#!/usr/bin/env bash

fallback_message="TEST MESSAGE - $HOSTNAME"
markdown_message="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <@U12345789>\n\`\`\`Can we do a\nmultiline code block\`\`\`"

# create markdown_message_unescaped with an unescaped version of markdown_message
printf -v markdown_message_unescaped %b "$markdown_message"

jq -n \
  --arg textMessage "$markdown_message_unescaped" \
  --arg fallbackMessage "$fallback_message" \
  --arg sectionType section --arg markdownType markdown '
    {
      text: $fallbackMessage, 
      blocks: [
        {
          type: $sectionType,
          text: {
                    type: $markdownType, 
                    text: $textMessage
                }
            }
        ]
    }'

...properly emits as output:

{
  "text": "TEST MESSAGE - YOUR_HOSTNAME",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "markdown",
        "text": "TEST MESSAGE - YOUR_HOSTNAME (0x0a) (\n)\nHi <@U12345789>\n```\nCan we do a multiline code block\n```"
      }
    }
  ]
}

Heller

I appreciate all your insight Charles, thank you very much! – Yah

This seems to work with bash but I'm seeing a weird discrepancy when using zsh: #73899610 (same without the printf %b actually) – Parma