How can I export my Firestore database from a Google Storage Bucket into a Json file

Question

Here's the situation: I have a Firestore Database. I download it on a daily basis to a Google Cloud Storage Bucket as a backup. If I want to download it locally, I do it using this command gsutil -m cp -r gs://BUCKET_PATH "DESTINATION_PATH" and it works fine.

MY PROBLEM: the format of the Bucket I download is LevelDB (I think). On my machine, it looks like this :

For example, this is my /users collection in Firestore when I download it through Google Storage. In the folder, I have multiple binary files ("output-...") and a file ( here: "all_namespaces_kind_users") for metadata.

MY GOAL: I want to be able to read my database in a json file.

MY TRIES:

• I try to use this convertor : https://github.com/Venryx/firestore-leveldb-tools but it uses Python2 and some old google libraries. Using this convertor, I have to download the SDKs locally (see below).

#repoRoot = os.getcwd()
repoRoot = os.path.dirname(os.path.realpath(__file__))

# import google sdks
sys.path.append(os.path.join(repoRoot, 'SDKs/google_appengine'))
sys.path.append(os.path.join(repoRoot, 'SDKs/google-cloud-sdk/lib/third_party'))
from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore

• Using the precedent convertor, I tried updating it to Python3, using these recommandations : https://github.com/Venryx/firestore-leveldb-tools/issues/1#issuecomment-674563473. I succeed to replace the first import from google.appengine.api.files import records but not the two others.
• For the two others : from google.appengine.datastore import entity_pb and from google.appengine.api import datastore, I saw this : https://pypi.org/project/protobuf-cloud-datastore-translator/ but I dont understand it fully.

MY QUESTION: How can I convert my Firestore Database (stored in a Google Bucket) to a json file ? Maybe someone has a complete different approach to propose ? Or should I stick to my way ? In that case, how can I finish to translate the convertor to Python3 using updated google librairies ?

BONUS QUESTION: How the h**l does not Google has a clean solution for that ? (or am I stupid?)

Strategist

What command or recipe are you using to backup the database? – Moina

I'm using a Cloud function and Google Cloud Scheduler to call the function everyday. I took the same Cloud function as written here : medium.com/swlh/… (under index.js) – Strategist

Hello, a good option could be that in your cloud function each time that you made a backcup to your bucket at the same time you export your database to json format using node-firebase-import-export (check the section exporting) and then you send it to save in other place. – Minne

Have you found any solution to this problem? I have the feeling Google makes this as painful as possible to get to your data to bind you their services – Separative

Just check the accepted solution. Works pretty well (except for subcollections). – Strategist

gsutil no longer maintained, went with gcloud storage cp -r gs://... – Potpie

Answer 1

I have created a converter in Python 3 which could convert firestore export files into JSON files firestore-export-json. The package provides a simple CLI command to covert the file.

fs_to_json [path_to_source_dir] -d [path_to_destination]

Lunette

I tested your script and add a pull request to it. I also found an issue. – Strategist

Unfortunately as of today this still doesn't work with a full backup, just with single collections – Separative

@Separative amazing tool, thanks a ton!! I was also successful in running it per-collection. Cheers – Washday