Algolia - get mass records & delete with filter

Question

I am using Algolia for search purposes and we got a huge pile of records. We want to delete some records and have decided to delete records that are older than X date.

First I was using this

const records = [];
const deleteRecordsBeforeDateAlgolia = (date) => {
    let client;

    *****

    const index = client.initIndex('function_message');
    //get the records before the given date
    try {
        index.search('',{
            filters: `time_stamp < ${date}`
        }).then(({hits}) => {
            if(hits.length > 0) {
                for (const hit of hits) {
                    index.deleteObject(hit.objectID);
                    records.push(hit.objectID);
                }
            }
            if(hits.length === 0) {
                console.log(`Deleted ${records.length} records`);
            } else {
                deleteRecordsBeforeDateAlgolia(date);
            }
        });
    } catch (err) {
        console.error(err);
    }
};

but I realized this isnt that optimized + will be very slow when deleting on prod. Can you tell me how I can get huge amounts of data with a filter (timestamp in this case) and then delete all of them?

EDIT

const records = [];
const deleteRecordsBeforeDateAlgolia = (date) => {
    let client;

    //creds stuff


    const index = client.initIndex('function_message');
    //get the records before the given date
    try {
        const search =  index.browseObjects({
            filters: `time_stamp < ${date}`
        }).then(res => {
            //IT SHOWS RESPONSE IS UNDEFINED
            res.forEach(record => {
                records.push(record);
            });
            console.log(`found ${records.length} records`);
        });
    } catch (err) {
        console.error(err);
    }
};

Bik

you can do bulk delete with deleteObjects. btw why not use async await. – Sapphera

wanted it to be synchronous thats why (i know the code is quite bad) – Bik

yeah what i was thinkinh with async await was something like const hits = await index.search.... and then idArray = hits.map... to return an array of objectID and then await deleteObjects(idArray) – Sapphera

the problem is, index.search will only result 20 results AT MAX. – Bik

have you tried the same thing in your current implementation instead of the for loop. map the hits and get id array and then do deleteObjects – Sapphera

How many records are you looking to retrieve? For a large number, you may be better off doings a "browse" than a "search", then pairing that with deleteObjects as @Sapphera recommends above. – Ret

@ChuckMeyer browse doesnt let me apply filters, or if it does, can you show how to? – Bik

@VineshRajpurohit didnt something like this work index.browseObjects({ query: '', filters: 'updated_at>1641905859', batch: batch => {...... – Sapphera

Check the edit please @Sapphera – Bik

@VineshRajpurohit try adding an empty query with query: '' and in the batch funtion you can do the pushing. this example algolia.com/doc/api-reference/api-methods/browse/… – Sapphera

Answer 1

browseObjects takes a batch callback function that's called on every batch of hits where you can specify what to do with the batch. The optional parameter list can be found here

Something like this should work

const records = [];

const deleteFromIndex = (idArray,index) => {
    index.deleteObjects(idArray).then(({ objectIDs }) => {
        console.log(objectIDs);
    });
}


const deleteRecordsBeforeDateAlgolia = (date) => {
  let client;
  client = algoliasearch('algoliaApp', 'algoliaKey');

  const index = client.initIndex('function_message');
  try {
    index.browseObjects({
        filters: `time_stamp<${date}`,
        query: '',
        batch: (batch) => {       
          records.push(...batch);  //push each batch into records array
        }
      })
      .then(() => {
        const idArray = records.map(({objectID}) => objectID) //get an array of objectIDs
        deleteFromIndex(idArray, index)
      });
  } catch (err) {
    console.error(err);
  }
};

deleteRecordsBeforeDateAlgolia('some date')

Sapphera