How to decode value in redis cli or shell script

Question

I've written redis-cli bash script to process all key and value for bulk retrieval but values are not printing as expected. When I give my key in redis-cli its printing with all special characters:

My Key and output from redis-cli

redis-cli MGET "0124" "0016"
1) "\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x00\x1d\x8e1\x0e\xc3@\b\x04?\x04\x0e8Q\x17\xa9\xf9\xdc\xdeY\x0b2\x91[\xfd>K\x99\xfd\xaf\xfc\x03\xeb-1\x10\xef\x00\x00\x00"
2) "\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x00\x1d\x8e1\x92\x031\b\x04?\x04k\x84\x10\xa0\xf8\x1;\xa8-7\xbb\xa2> \xc0n\xdc\xe1\xce\xdb\xbdk\xac\x81\x9a]Q*\x8ex\xa4\xe0\x99\xd5\xd1\xb3\x94w^\x9f]\xa7$2\xce;\xdcp\x9c\x9b\xff;\xff\x01\xb3\xcc\xd5H\xf0\x00\x00\x00"

Can someone please help how to decode this value in redis-cli or shell script. I'm new to Redis - it would be really appreciated if you can help me to fix this issue

I used gunzip - but I'm getting below error:

redis-cli -h GET "100" | gunzip

Error:

   gzip: stdin: unexpected end of file
   gzip: stdin: decompression OK, trailing garbage ignored

Redis-cli code:

#!/bin/sh
hostName=$1
port=$2
count=$3
cursor=-1
keys=""
recordCount=0
while [ $cursor -ne 0 ];
do
        if [ $cursor -eq -1 ]
        then
        cursor=0
    fi
    reply=`redis-cli -h $hostName -p $port SCAN $cursor MATCH "*" COUNT $count`
    cursor=`expr "$reply" : '\([0-9]*[0-9 ]\)'`
    keys=${reply#[0-9]*[[:space:]]}
    value=$(redis-cli -h $hostName -p $port MGET $keys)
    temCount=`echo $value | awk -F\| '{print NF-1}'`
    recordCount=`expr ${temCount} + ${recordCount}`
done
echo "Total no. of documents are: " $recordCount

My Redis key-value pattern:

Keys - 123.item.media
Values - 93839,abc,98,829|38282,yiw,282,282|8922,dux,382,993|

Keys - 234.item.media
Values - 2122,eww,92,211|8332,uei,902,872|9039,uns,892,782|

Keys - 839.item.media
Values - 7822,nkp,77,002|7821,mko,999,822|

Tartar

echo -e "..." should do the trick. – Mnemonics

I tried something like this value=$(redis-cli MGET $keys) echo -e $value | gunzip . But it didn't work – Tartar

What about value=echo -e "$(redis-cli MGET $keys)" | gunzip ? – Mnemonics

I'm getting -e: command not found – Tartar

sorry, value=$(echo -e "$(redis-cli MGET $keys)" | gunzip) – Mnemonics

It says gzip: stdin: unexpected end of file :( – Tartar

Let us continue this discussion in chat. – Tartar

@Mr.bug, echo is not permitted to have -e do anything but print the two-character sequence -e on output by the POSIX standard (this is one of very few places where bash is actually doing something the standard prohibits, rather than putting extensions in undefined space), and in some modes (when both xpg_echo and posix flags are set), bash doesn't offer said extension. Consider making a habit of using printf '%b\n' "$value" instead. See also pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html, particularly the APPLICATION USAGE and RATIONALE sections. – Hazan

Answer 1

TL;DR: You'll need to convert the escaped binary strings back to binary to decode.

An older question, but still valid today. The key's value contains binary, so Redis is showing you an escaped version of the string to make it printable in the terminal.

The non-printable characters are notated with the \x and then two hex character representing the non-printable 8 bits. We can also tell that the values in your example are gzip encoded, because the magic number prefix of \x1f\x8b (aka 1f 8b).

Example to recreate:

First, create a new binary blob, we'll use gzip again for the example (-n means no tailing newline character):

echo -n "hello world" | gzip > test.gz

If we cat the binary, the shell can't render it properly and it'll look like this:

cat test.gz
�%f�H���W(�/�I�J

Instead, lets use python, to view the escaped binary string (note the b prefix in the result tells us it's a binary string):

python3 -c "print(open('test.gz','rb').read())"
b'\x1f\x8b\x08\x00\xa1%\x0ff\x00\x03\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00'

Next we send that binary (gzip data) blob to Redis (-x to read from stdin):

redis-cli -x set example-key001 < test.gz
OK

If we get our key now, we'll see the familuar escaped string that python printed above:

redis-cli get example-key001
"\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00"

To retrieve the original uncompressed text value from the key, we can get the raw key and pass value directly to gzip to decompress (I added tr here as redis-cli --raw still appends newlines):

redis-cli --raw get example-key001 | tr -d '\n' | gzip -d
hello world

---

PS in the comments Mr. bug mentioned the echo -e and pointed out this isn't part of posix. Unfortunately, the link to the chat is long since broken to see the rest of the conversation, but he's also correct. echo is a shell built in, so using #!/bin/sh means that there is no echo -e. You can use #!/usr/bin/env bash to use bash, and it's more portable than assuming bash is always installed in /bin.

Here's an example of what he was saying:

#!/usr/bin/env bash

echo -n -e '\x1f\x8b\b\x00\x00\x00\x00\x00\x00\x03\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00' > bin.gz
gzip -d < bin.gz
hello world

Sparry