jq : Generate UUID in field

Question

I have a requirement to tag records uniquely with UUIDs (for a correlation id). I cant see a direct way to do this via the options, is such a thing possible? If not, is there some kind of workaround that might be able to do this?

Is it even possible to generate a random number or string in jq?

Achlamydeous

Answer 1

jq currently has no support for UUID generation, so your best bet would be to feed UUIDs in to jq, e.g. along these lines:

ruby -e 'require "securerandom"; p SecureRandom.uuid' | jq '{uuid: .}'
{
  "uuid": "5657dd65-a495-4487-9887-c7f0e01645c9"
}

The PRNG contributions for jq have unfortunately not yet made their way into an official release. For examples of PRNG generators written in jq, see e.g. rosettacode:

https://rosettacode.org/wiki/Linear_congruential_generator#jq

Reading from an unbounded stream of UUIDs

Assuming the availability of a uuid generator such as uuidgen, you could use input or inputs along the following lines:

jq -nR '[range(0;10) | input]' < <(while true; do uuidgen ; done)

(Notice that an OS pipe has been avoided here.)

Palstave

Thanks, but in my case I have a very large json file with many separate objects, and I need to enrich each one with a uuid. I can't feasibly slurp all the data into memory, and I don't necessarily know how many json objects are in the file before processing. Is there a way I can generate an infinite stream of uuids and join it to each json object using jq without slurping? – Achlamydeous

I'd read the JSON using --argfile (or maybe --slurpfile), and read the unbounded stream of UUIDs using input or inputs, as shown in the updated answer. – Palstave

Answer 2

It is possible to generate pseudo-random numbers in jq if you provide one initial random number (--argjson initialRandomNumber). Passing $RANDOM$RANDOM instead of $RANDOM is intended to increase the range for the initial pseudo-random value.

I used a slightly modified version of the function nextRandomNumber from Rosettacode jq: random numbers to generate random numbers, Strings and UUIDs as shown in the following code.

Each function takes a parameter $state and delivers a newState in the response for a subsequent call.

Because you ask how to generate a

1. random number
2. random string
3. UUID

you can find 6 functions in my code to generate a single instance and an array of them.

You can pick the function you need and use it as a filter in jq.

#!/bin/bash

jq -c -r -n --argjson initialRandomNumber "$RANDOM$RANDOM" '
  # 15-bit integers generated using the same formula as rand() from the Microsoft C Runtime.
  # The random numbers are in [0 -- 32767] inclusive.
  #
  # Input: 
  #   first call:      $state = a random number provided to jq by parameter
  #   subsequent call: $state = "newState" from last response
  #
  # Output: 
  #   object with pseudo-random number and "newState" for a subsequent call.
  def nextRandomNumber($state):
    ( (214013 * $state) + 2531011) % 2147483648 # mod 2^31
    | { newState: .,
        randomNumber: (. / 65536 | floor) };

  def nextRandomNumbers($state; $count):
    [foreach range($count) as $x (nextRandomNumber($state); nextRandomNumber(.newState); .)]
    | { newState: .[-1].newState,
        randomNumbers: map(.randomNumber) };


# ----- random UUID ---------------

  def hexByte:
    [. / 256 % 16, . % 16]
    | map(if . < 10 then . + 48 else . + 87 end)   # ASCII: 0...9: 48-57, a...f: 97-102
    | implode;

  def nextRandomUUID($state):
    nextRandomNumbers($state; 16)
    | .newState as $newState
    | .randomNumbers
    | map(hexByte)
    | "\(.[0:4] | join(""))-\(.[4:6] | join(""))-\(.[6:8] | join(""))-\(.[8:10] | join(""))-\(.[10:] | join(""))"
    | { newState: $newState,
        randomUUID: . };

  def nextRandomUUIDs($state; $count):
    [foreach range($count) as $x (nextRandomUUID($state); nextRandomUUID(.newState); .)]
    | { newState: .[-1].newState,
        randomUUIDs: map(.randomUUID) };


# ----- random String ---------------

  def letter:
    . % 52
    | [if . < 26 then . + 65 else . + 71 end]   # ASCII: A...Z: 65-90, a...z: 97-122
    | implode;

  def nextRandomString($state; $minLength; $maxLength):
    nextRandomNumber($state)
    | (try (.randomNumber % ($maxLength - $minLength + 1) + $minLength) catch $minLength) as $length
    | nextRandomNumbers(.newState; $length)
    | .newState as $newState
    | .randomNumbers
    | map(letter)
    | join("")
    | { newState: $newState,
        randomString: . };

  def nextRandomStrings($state; $count; $minLength; $maxLength):
    [foreach range($count) as $x (nextRandomString($state; $minLength; $maxLength); nextRandomString(.newState; $minLength; $maxLength); .)]
    | { newState: .[-1].newState,
        randomStrings: map(.randomString) };


# ----- example usage ---------------

  nextRandomNumber($initialRandomNumber)               # see output 1
# nextRandomNumbers($initialRandomNumber; 3)           # see output 2
# nextRandomUUID($initialRandomNumber)                 # see output 3
# nextRandomUUIDs($initialRandomNumber; 3)             # see output 4
# nextRandomString($initialRandomNumber; 10; 15)       # see output 5
# nextRandomStrings($initialRandomNumber; 3; 6; 10)    # see output 6
# nextRandomNumber($initialRandomNumber) | nextRandomNumbers(.newState; 3)   # see output 7
'

Outputs

output 1: generate pseudo-random number

{"newState":912028498,"randomNumber":13916}

---

output 2: generate 3 pseudo-random numbers

{"newState":677282016,"randomNumbers":[10202,20943,6980]}`

---

output 3: generate random UUID

{"newState":1188119770,"randomUUID":"cdcda95b-af57-1303-da72-d21c6e7b1861"}

---

output 4: generate 3 random UUIDs

{"newState":907540185,"randomUUIDs":["855c1445-b529-4301-a535-20cb298feaff","5b685e49-8596-830e-f56a-0a22c43c4c32","35fed6d8-d72b-2833-fd6f-f99154358067"]}

---

output 5: generate random Strings with length 10-15

{"newState":1037126684,"randomString":"SJadqPGkERAu"}`

---

output 6: generate 3 random Strings with length 6-10

{"newState":316121190,"randomStrings":["eNKxechu","XPkvNg","TIABHbYCxB"]}`

---

output 7: using newState for a second call to generate 3 random numbers

{"newState":808494511,"randomNumbers":[26045,16811,12336]}`

Birdsall

Answer 3

If using Linux (MacOS?), you can use the content of /proc/sys/kernel/random/uuid:

❯ jq -n --arg uuid "$(cat /proc/sys/kernel/random/uuid)"  '{"id": $uuid}'

{
  "id": "26b8054a-9020-4290-8e77-f52a86e3933e"
}

Afrika

Answer 4

Old question, but I recently ran into the same problem and came up with a somewhat general way of handling this. You could combine jq's compact format (-c option) and read each item in shell, generate a uuid externally, then feed that to jq again to insert it. Also, if you're handling really large files, chances are you might benefit from jq's streaming mode (--stream option). Here's an example (using uuidgen) if you have a big array file:

jq --stream -cn 'fromstream(1 | truncate_stream(inputs))' big-array.json | while read -r item; do
  jq '.id |= $uuid' --arg uuid "$(uuidgen)" <<<"$item"
done

You could also slurp the output to get the array back with jq -s . if needed.

Donative