How to read Base64 VLQ code?

Question

I'm trying to understand how css source map works. I've created a very simple scss file.

#navbar {
    color: black;
}

When I compile the above scss, I get the following map file.

{
"version": "3",
"mappings": "AAAA,OAAQ;EACP,KAAK,EAAE,KAAK",
"sources": ["test.scss"],
"file": "test.css"
}

when I decode "mappings", I get the following values.

0) [0,0,0,0], [7,0,0,8]
1) [2,0,1,-7], [5,0,0,5], [2,0,0,2], [5,0,0,5]

What are those values?

Egide

Answer 1

I found an example at http://www.thecssninja.com/javascript/source-mapping, under the section "Base64 VLQ and keeping the source map small".

The above diagram AAgBC once processed further would return 0, 0, 32, 16, 1 – the 32 being the continuation bit that helps build the following value of 16. B purely decoded in Base64 is 1. So the important values that are used are 0, 0, 16, 1. This then lets us know that line 1 (lines are kept count by the semi colons) column 0 of the generated file maps to file 0 (array of files 0 is foo.js), line 16 at column 1.

Forayer

Answer 2

Even after reading the answers, explanations were still not so clear to me. Here is an explanation in plain english in case it helps someone:

something like ;;AAAA,IAAM,WAAW,SAAX;... means <line0 info>;<line1 info>;...

so for ;;AAAA;IAAM,WAAW,SAAX;..., line 0 and line 1 doesn't have any important info (empty spaces etc.)

then for line 2 we have AAAA,IAAM,WAAW,SAAX

we convert each of these groups to binary using the base64 character mapping:

BASE64_ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

so we basically find the index in this BASE64_ALPHABET above, and convert the index to 6-bit binary (6 bit because we use base64). eg. index of A is 0, so in 6 bit binary its 000000. so AAAA would be 000000 000000 000000 000000

then if we do this with IAAM we get: 001000 000000 000000 001100.

then this bit representation is the VLQ encoded version of 4 numbers. we start from the left block, remove the sign and continuation bit, keep the bits. and continue adding it bits while continuation bit is 1.

eg. 001000 is (cont)0100(sign)
so cont = 0 (no other block will be added to this number)
sign=0 (its positive)
bits = 0100 --> so it is 4 in decimal

-- note that we only remove sign bit for the first time. so if we had
101000 001000
we would say
0100 (cont=1, sign=0) 01000 (cont=0)
so we would have had +010001000 = 136

when we keep doing this, we will get these 4 numbers (continuation bit should be 0 exactly 4 times).

• AAAA would map to (0,0,0,0)
• IAAM would map to (4,0,0,6)
• WAAW would map to (11,0,0,11) ...

now, each of these mean relative numbers. so we correct those:

• AAAA actually points to: (0,0,0,0)
• IAAM actually points to: (0+4, 0+0, 0+0, 0+6) = (4,0,0,6)
• WAAW actually points to: (4+11, 0+0, 0+0, 6+11) = (15,0,0,17) // we added it where IAAAM was actually pointing to

...

so numbers (n1, n2, n3, n4) here stand for

• n1: column in generated code
• n2: corresponding source file index in "sources" array of sourceMapping output
• n3: line number in original code
• n4: column number in original code

we already knew which line this referred to from the beginning. so using the information we have above, we learned:

• AAAA: line 2, column 1 of generated code points to sources[0], line 0, column 0
• IAAM: line 2, column 4 of generated code points to sources[0], line 0, column 6
• WAAW: line 2, column 15 of generated code points to sources[0], line 0, column 17 ...

two good sources about this:

• more on VLQ encoding
• more on how to interpret/decode

Antrorse

Answer 3

Despite the examples I could find, I took me quite a while to understand how the coding / decoding really works. So I thought I'd learn best by trying to make something myself in a very explicit, step by step way. I started out with the explanation of VLQ at this blog,

I use the following Python functor to generate sourcemaps for Transcrypt. The code is simple and I think gives good insight in how the coding/decoding works in principle. To achieve speed despite its simplicity, it caches the first 256 numbers, which are used most often in generating a v3 sourcemap.

import math

class GetBase64Vlq:
    def __init__ (self):
        self.nBits32 = 5
        self.encoding = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
        self.prefabSize = 256
        self.prefab = [self (i, True) for i in range (self.prefabSize)]

    def __call__ (self, anInteger, init = False):
        if not init and 0 < anInteger < self.prefabSize:
            return self.prefab [anInteger]
        else:
            signed = bin (abs (anInteger)) [2 : ] + ('1' if anInteger < 0 else '0')
            nChunks = math.ceil (len (signed) / float (self.nBits32))
            padded = (self.nBits32 * '0' + signed) [-nChunks * self.nBits32 : ]
            chunks = [('1' if iChunk else '0') + padded [iChunk * self.nBits32 : (iChunk + 1) * self.nBits32] for iChunk in range (nChunks - 1, -1, -1)]
            return ''.join ([self.encoding [int (chunk, 2)] for chunk in chunks])

getBase64Vlq = GetBase64Vlq ()

Example of use:

while (True):
    print (getBase64Vlq (int (input ('Give number:'))))

Camilia

Answer 4

based on Jacques answer, I made the following JS snippet to parse a segment into it corresponding integers fields.

const VLQ_BASE = 5;
const VLQ_CONT = 1 << VLQ_BASE;  // 100000
const VLQ_MASK = VLQ_CONT - 1;   // 011111
const b64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
function vlq(segment) {
    const fields = [];
    for (let segIdx = 0; segIdx < segment.length;) {
        let result = 0;
        for (let shift = 0; ; shift += VLQ_BASE) {
            const digit = b64.indexOf(segment.charAt(segIdx++));
            result += ((digit & VLQ_MASK) << shift);
            if (!(digit & VLQ_CONT)) break;
        };
        fields.push(result & 1 ? -(result >> 1) : result >> 1)
    }
    return fields; // col,srcIdx,lineSrc,colSrc,nameIdx
};

this allows me to lookup the original line given a current line

    const fields = sourceMap.mappings.split(';').map(group=>group.split(',').map(vlq));
    let sum = 0;
    return fields.map(f => sum += f[0][2])[JsLine];

Married