Trying to simulate cell level TTL in bigtable but whole column family data is getting removed by garbage collection

Question

created a table with the following rules: so with this, data should expire after 1 second (as per docs)

async function createTable() {
    console.log("Creating Table");
    const options = {
        families: [
            {
                name: 'payloads',
                rule: {
                    age: {
                        seconds: 1,
                        nanos: 0,
                    },
                },
            },
        ],
    };
    try {
        await table.create(options);
        console.log("Successfully Created Table");
    } catch (err) {
        console.error(`Error reading rows :`, err);
    }
}

And then inserted the data like this:

      const rowsToInsert = {
            key: "SUMEET",
            data: {
                payloads: {
                    '1': {
                        value: "NOTIFICATIONS_PAYLOAD_1",
                        timestamp: 1576500927000,
                    },
                    '2': {
                        value: "NOTIFICATIONS_PAYLOAD_2",
                        timestamp: 1576587327000,
                    },
                    '3': {
                        value: "NOTIFICATIONS_PAYLOAD_3",
                        timestamp: 1576673727000,
                    },
                    '4': {
                        value: "NOTIFICATIONS_PAYLOAD_4",
                        timestamp: 1576760127000,
                    },
                },
            },
        };

        await table.insert(rowsToInsert);

so i added four cells with different timeStamp:

1. First with 5 minutes ahead of time when am writing the data
2. Second with 1 hour ahead
3. third with 1 day ahead
4. fourth with 2 day ahead

problem here is when m reading the data whole column family data is getting deleted but it should delete only first and second cell only as per the rules set

Is there anything am missing or am doing wrong ??

Fredra

Answer 1

The issue is in your timestamp, you are most likely setting dates from the past. I would recommend you to set the dates with the date methods from javascript instead of manually setting them up like you are doing right now.

I did some tests with following code:

const Bigtable = require('@google-cloud/bigtable');
const bigtable = Bigtable();
const instance = bigtable.instance([instance]);
const table = instance.table([table]);
const now = new Date();


async function writeSimple() {
    var now = new Date(Date.now());
    var fiveMinutes = new Date(now.getTime() + 5 * 60000);
    var anHour = new Date(now.getTime() + 60 * 60000);
    var aDay = new Date(now.getTime() + 24 * 60 * 60000);
    var twoDays = new Date(now.getTime() + 48 * 60 * 60000);
  const rowsToInsert = {
            key: "SUMEET",
            data: {
                payloads: {
                    '1': {
                        value: "NOTIFICATIONS_PAYLOAD_1",
                        timestamp: now,
                    },
                    '2': {
                        value: "NOTIFICATIONS_PAYLOAD_2",
                        timestamp: fiveMinutes,
                    },
                    '3': {
                        value: "NOTIFICATIONS_PAYLOAD_3",
                        timestamp: anHour,
                    },
                    '4': {
                        value: "NOTIFICATIONS_PAYLOAD_4",
                        timestamp: aDay,
                    },
                    '5': {
                        value: "NOTIFICATIONS_PAYLOAD_5",
                        timestamp: twoDays,
                    },
                },
            },
        };

        await table.insert(rowsToInsert);
        console.log(`Successfully wrote row ${rowsToInsert.key}`);
}

and obtained a row like the following one:

2019/12/17 16:53:33 -creds flag unset, will use gcloud credential
----------------------------------------
SUMEET
  payloads:1                               @ 2019/12/17-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_1"
  payloads:2                               @ 2019/12/17-16:35:34.343000
    "NOTIFICATIONS_PAYLOAD_2"
  payloads:3                               @ 2019/12/17-17:30:34.343000
    "NOTIFICATIONS_PAYLOAD_3"
  payloads:4                               @ 2019/12/18-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_4"
  payloads:5                               @ 2019/12/19-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_5"

And after the garbage collector passed (around 15 minutes later) I got the result that you desired:

2019/12/17 16:59:47 -creds flag unset, will use gcloud credential
----------------------------------------
SUMEET
  payloads:3                               @ 2019/12/17-17:30:34.343000
    "NOTIFICATIONS_PAYLOAD_3"
  payloads:4                               @ 2019/12/18-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_4"
  payloads:5                               @ 2019/12/19-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_5"

I hope you find this useful!

Rolon

Glad to have been of help :) – Rolon

Answer 2

You have the timestamp in milliseconds. Cloud Bigtable requires microseconds (see here). Basically, add '000' to the end of all of your timestamps.

Schuler