Maintain data in Google Bigtable for longer periods

Question

We have use-cases where we would like to store a large volume of data in Google Bigtable for long periods:

• during product development
• for performance tuning
• for demos

We need to store the data but we don't really need it to be "online" all the time. The current cost bottleneck seems to be the cost of nodes which in these cases are idle for long periods.

How is Google Bigtable being used during product development? I am aware of the development mode (and the emulator) and they are fine for some use-cases but we still need the production environment for other use-cases.

Really, what would be ideal is the ability to switch "off" Bigtable (while still paying for data stored but not for nodes) and bring up the nodes when needed. I don't believe this feature exists. In its absence are there other possible workarounds/alternatives?

Librate

Out of curiosity, what happens if you disable billing for the project in question? I know this works for what you are trying to do for other types of GCP projects. – Rioux

@DominicTracey: Thanks - I am trying that out. The message on disabling billing is not very specific though. After disabling, my Bigtable instance is still "there", Although this is certainly promising, I am concerned that at some point it is going to be deleted. I'll keep an eye out and post back what I find. – Librate

How much data are you storing? I guess re-importing data from a backup is not feasible? When you need to do a demo, do you need read-write or read-only access? Would you be serving live-traffic to users or just need to do analytics? You're right that such a feature is not yet available, just trying to figure out what parameters are important for potential workarounds. – Jabiru

@MishaBrukman: Between 9-10TB uncompressed (best we can do right now) and hope to increase that to better represent the scale we want to support. New to Bigtable - but we are calculating if loading from a backup is cost-effective and practical. Seems to be a function of idle time, data-size and time to reload but doesn't look promising. For demos - largely read-only access (any writes - I guess we can work around). We hope Bigtable can be the data store for our application - so it will serve live traffic when in production. – Librate

@DominicTracey: An update - the instance has been restored when billing was re-enabled (after a day and a half). Still a concern that we will hit some unknown time-limit where the instance is removed - especially if it is a particularly large instance. This definitely seems like a possible workaround if we could get some guidelines around the time frame of retention. Thanks! – Librate

Also agree that some form of 'idling' or a free tier would be great, particularly for smaller organizations. In the end this should help Google, since smaller companies would have a better chance 'selling' BigTable as a solution to their own clients. We are in marketing mode right now and can't keep even the smallest cluster up continuously, as even $2-3 per hour adds up. As a workaround we had to build a smaller SQL-based demo and 'promise' to the audience that we have an equivalent BibTable version, but that looks far less credible... – Vedi

Answer 1

It's an interesting question. I've done it with smaller projects using Datastore with much smaller sizes (~2Gb) that have hung around for years after disabling billing. Given how much it costs to do backup/restores on those projects, I could imagine this would be cost prohibitive solution in the BigTable world. It is disappointing that Google hasn't provided a better solution for this. They do talk about different storage classes so I'd imagine disabling a project would move its assets to coldline - but that is just rank speculation on my part.

Rioux

I definitely think this is a possible workaround. I am just going to let the question hang around a little while longer to see if there are any other suggestions (and also waiting on Misha :-)) - otherwise I will mark this as an answer. – Librate