Cloud function to export Firestore backup data. Using firebase-admin or @google-cloud/firestore?

Question

I'm currently trying to build a cloud function to export my Firestore data to my Storage Bucket.

The only example I've found on the Firebase DOCs on how to do this:

https://googleapis.dev/nodejs/firestore/latest/v1.FirestoreAdminClient.html#exportDocuments

EXAMPLE

const firestore = require('@google-cloud/firestore');

const client = new firestore.v1.FirestoreAdminClient({
  // optional auth parameters.
});

const formattedName = client.databasePath('[PROJECT]', '[DATABASE]');
client.exportDocuments({name: formattedName})
  .then(responses => {
    const response = responses[0];
    // doThingsWith(response)
  })
  .catch(err => {
    console.error(err);
  });

From that example, it seems that I need to install @google-cloud/firestore as a dependency to my cloud function.

But I was wondering if I can access these methods using only the firebase-admin package.

I've thought of that because the firebase-admin has the @google-cloud/firestore as a dependency already.

> firebase-admin > package.json

"dependencies": {
    "@firebase/database": "^0.4.7",
    "@google-cloud/firestore": "^2.0.0",    // <---------------------
    "@google-cloud/storage": "^3.0.2",
    "@types/node": "^8.0.53",
    "dicer": "^0.3.0",
    "jsonwebtoken": "8.1.0",
    "node-forge": "0.7.4"
  },

QUESTION:

Is it possible to get an instance of the FirestoreAdminClient and use the exportDocuments method using just the firebase-admin ?

Or do I really need to install the @google-cloud/firestore as a direct dependency and work with it directly?

Plosive

Answer 1

The way you're accessing the admin client is correct as far as I can tell.

const client = new admin.firestore.v1.FirestoreAdminClient({});

However, you probably won't get any TypeScript/intellisense help beyond this point since the Firestore library does not actually define detailed typings for v1 RPCs. Notice how they are declared with any types: https://github.com/googleapis/nodejs-firestore/blob/425bf3d3f5ecab66fcecf5373e8dd03b73bb46ad/types/firestore.d.ts#L1354-L1364

Sailor

It does work! No intellisense (as per the reason you've pointed out). But it works. Thanks. – Plosive

You can do something like this to get TypeScript/intellisense work (TypeScript only, sorry Flow users :/) import { FirestoreAdminClient } from '@google-cloud/firestore/build/src/v1/firestore_admin_client'; const FirestoreV1AdminClient = admin.firestore.v1.FirestoreAdminClient as typeof FirestoreAdminClient; – Catchment

Answer 2

Here is an implementation I'm using that allows you to do whatever operations you need to do, based on the template provided by firebase here https://firebase.google.com/docs/firestore/solutions/schedule-export

In my case I'm filtering out collections from firestore I don't want the scheduler to automatically backup

const { Firestore } = require('@google-cloud/firestore')

const firestore = new Firestore()
const client = new Firestore.v1.FirestoreAdminClient()
const bucket = 'gs://backups-user-data'

exports.scheduledFirestoreBackupUserData = async (event, context) => {
  const databaseName = client.databasePath(
    process.env.GCLOUD_PROJECT,
    '(default)'
  )

  const collectionsToExclude = ['_welcome', 'eventIds', 'analyticsData']

  const collectionsToBackup = await firestore.listCollections()
    .then(collectionRefs => {
      return collectionRefs
        .map(ref => ref.id)
        .filter(id => !collectionsToExclude.includes(id))
    })


  return client
    .exportDocuments({
      name: databaseName,
      outputUriPrefix: bucket,
      // Leave collectionIds empty to export all collections
      // or define a list of collection IDs:
      // collectionIds: ['users', 'posts']
      collectionIds: [...collectionsToBackup]
    })
    .then(responses => {
      const response = responses[0]
      console.log(`Operation Name: ${response['name']}`)
      return response
    })
    .catch(err => {
      console.error(err)
    })
}

Preponderant

Note it seems on Node10 runtime the variable process.env.GCLOUD_PROJECT is not available by default and the script won't run. – Stefanysteffane

@Vojtěch could be the case, yes, as I currently have my functions running still on Node 8... if you find a fix please let me know as well. – Preponderant

i just pass the cloud project id as env var manually and it works, – Stefanysteffane

You directly return the async function and the function finishes instantly, even though the export operation is still running. Shouldn't be there some kind of return await client.exportDocuments(... to prevent that behaviour? – Stefanysteffane

simply add GCLOUD_PROJECT with your projectId as an environment variable while defining your function. – Crusade

Answer 3

Here is the full explanation with code (I use it and it works very well) on how to do automated Firestore backups by mixing Cloud Scheduler, PubSub and Firebase Function https://firebase.google.com/docs/firestore/solutions/schedule-export

Penitential

Answer 4

firebase-admin just wraps the Cloud SDK and re-exports its symbols. You can use the wrapper, or use the Cloud SDK directly, or even a combination of the two if you want. If you want to use both, you have to declare an explicit dependency on @google-cloud/firestore in order to be able to import it directly into your code.

Crossroad

If you could, would you mind showing me an example of how to get a hold of the FirestoreAdminClient using the firebase-admin ? I've tried: const client = new admin.firestore.v1.FirestoreAdminClient({}); and it was all auto-completed by VSCode. But I'm not getting any auto-completion on my client variable after that. Am I doing something wrong? Thanks! – Plosive

I've never used FirestoreAdminClient. Typical access is going to happen through the Firestore object. googleapis.dev/nodejs/firestore/latest/Firestore.html – Crossroad