How to retrieve more than 1000 rows from Parse.com?

Question

I have been using Parse to retrieve a data for a list view. Unfortunately they limit requests to 100 by default to a 1000 max. I have well over that 1000 max in my class. I found a link on the web which shows a way to do it on iOS but how would you do it on Android? Web Link

I am currently adding all the data into a arraylist in a loop until all items are complete (100) then adding them to the list

Rafaelita

Answer 1

I have figured out how to achieve my goal:

Declare Global Variable

private static List<ParseObject>allObjects = new ArrayList<ParseObject>();

Create Query

final ParseQuery parseQuery = new ParseQuery("Objects");
parseQuery.setLimit(1000);
parseQuery.findInBackground(getAllObjects());

Callback for Query

int skip=0;
FindCallback getAllObjects(){
    return new FindCallback(){
        public void done(List<ParseObject> objects, ParseException e) {
            if (e == null) {

                allObjects.addAll(objects);
                 int limit =1000;
                if (objects.size() == limit){
                    skip = skip + limit;
                    ParseQuery query = new ParseQuery("Objects");
                    query.setSkip(skip);
                    query.setLimit(limit);
                    query.findInBackground(getAllObjects());
                }
                //We have a full PokeDex
                else {
                    //USE FULL DATA AS INTENDED
                }
        }
    };
}

Rafaelita

What's the initial value of skip? – Crossruff

The initial value should be 0, as you want to start at the first record, and keep skipping the limit until you have all the objects – Rafaelita

i had to make 'skip' var to global for get this code working. but axcept for that code is greate – Regulus

Not sure why but it is not working for me. it's asking me to implement two: done(List objects, ParseException e) and done(Object o, Throwable throwable) methods and it is always calling the one with the one object – Paz

Answer 2

Here is a JavaScript version without promises..

These are the global variables (collections are not required, just a bad habit of mine)..

   ///create a collection of cool things and instantiate it (globally)
    var CoolCollection = Parse.Collection.extend({
       model: CoolThing
    }), coolCollection = new CoolCollection();

This is the "looping" function that gets your results..

//recursive call, initial loopCount is 0 (we haven't looped yet)
function getAllRecords(loopCount){

    ///set your record limit
    var limit = 1000;

    ///create your eggstra-special query
     new Parse.Query(CoolThings)
            .limit(limit)
            .skip(limit * loopCount) //<-important
            .find({
             success: function (results) {
                 if(results.length > 0){

                     //we do stuff in here like "add items to a collection of cool things"
                     for(var j=0; j < results.length; j++){
                         coolCollection.add(results[j]);
                     }

                     loopCount++; //<--increment our loop because we are not done

                     getAllRecords(loopCount); //<--recurse
                 }
                 else
                 {
                     //our query has run out of steam, this else{} will be called one time only
                     coolCollection.each(function(coolThing){
                        //do something awesome with each of your cool things
                     });
                 }
            },
             error: function (error) {
                //badness with the find
             }
         });
}

This is how you call it (or you could do it other ways):

getAllRecords(0);

Fireeater

Answer 3

JAVA

So after 5 years, 4 months the above answer of @SquiresSquire needed some changes to make it work for me, and I would like to share it with you.

private static List<ParseObject>allObjects = new ArrayList<ParseObject>();

ParseQuery<ParseObject> parseQuery = new ParseQuery<ParseObject>("CLASSNAME");
parseQuery.setLimit(1000);
parseQuery.findInBackground(getAllObjects());

FindCallback <ParseObject> getAllObjects() {
    return new FindCallback <ParseObject>() {
        @Override
        public void done(List<ParseObject> objects, ParseException e) {
            if (e == null) {
                allObjects.addAll(objects);
                int limit = 1000;
                if (objects.size() == limit) {
                    skip = skip + limit;
                    ParseQuery query = new ParseQuery("CLASSNAME");
                    query.setSkip(skip);
                    query.setLimit(limit);
                    query.findInBackground(getAllObjects());
                }
                //We have a full PokeDex
                else {
                    //USE FULL DATA AS INTENDED
                }
            }
        }
    
    };

Paz

Answer 4

IMPORTANT None of the answers here are useful if you are using open source parse server then it does limit 100 rows by default but you can put any value in query,limit(100000) //WORKS No need for recursive calls just put the limit to number of rows you want.

https://github.com/parse-community/parse-server/issues/5383

Ramtil

Answer 5

In C# I use this recursion:

private static async Task GetAll(int count = 0, int limit = 1000)
{
    if (count * limit != list.Count) return;
    var res = await ParseObject.GetQuery("Row").Limit(limit).Skip(list.Count).FindAsync();
    res.ToList().ForEach(x => list.Add(x));
    await GetAll(++count);
}

JS version:

function getAll(list) {
    new Parse.Query(Row).limit(1000).skip(list.length).find().then(function (result) {
        list = list.concat(result);
        if (result.length != 1000) {
            //do here something with the list...
            return;
        }

        getAll(list);
    });
}

Usage: GetAll() in C#, and getAll([]) in JS.

I store all rows from the class Rowin the list. In each request I get 1000 rows and skip the current size of the list. Recursion stops when the current number of exported rows is different from the expected.

Archilochus

Answer 6

**EDIT : Below answer is redundant because open source parse server doesn't put any limit on max rows to be fetched

 //instead of var result = await query.find();
    query.limit(99999999999);//Any value greater then max rows you want
    var result = await query.find();**

Original answer:

Javascript / Cloud Code

Here's a clean way working for all queries

async function fetchAllIgnoringLimit(query,result) {
  const limit = 1000;
  query.limit(limit);
  query.skip(result.length);

  const results = await query.find();
  result = result.concat(results)

  if(results.length === limit) {
    return await fetchAllIgnoringLimit(query,result );
  } else {
    return result;
  }

}

And here's how to use it

var GameScore = Parse.Object.extend("GameScore");
var query = new Parse.Query(GameScore);
//instead of var result = await query.find();
var result = await fetchAllIgnoringLimit(query,new Array());
console.log("got "+result.length+" rows")

Ramtil

Thanks for posting this version, but I think there's an error with your results.length === limit check because you have already reassigned result to the combined results. It might be safer all around to replace result = ... with a new variable, such as const combined = ... and pass that to the next call to fetchAllIgnoringLimit – Jacindajacinta

IMP - You should not use this above answer.Open source parse server doesnt restrict max 1000 rows See my other answer https://mcmap.net/q/568338/-how-to-retrieve-more-than-1000-rows-from-parse-com – Ramtil

Answer 7

YAS (Yet Another Solution!) Using async() and await() in javascript.

async parseFetchAll(collected = []) {
  let query = new Parse.Query(GameScore);
  const limit = 1000;

  query.limit(limit);
  query.skip(collected.length);

  const results = await query.find();

  if(results.length === limit) {
    return await parseFetchAll([ ...collected, ...results ]);
  } else {
    return collected.concat(results);
  }

}

Mackenie

Answer 8

A Swift 3 Example:

var users    = [String] ()
var payments = [String] ()
///set your record limit
let limit = 29
//recursive call, initial loopCount is 0 (we haven't looped yet)
func loadAllPaymentDetails(_ loopCount:  Int){
    ///create your NEW eggstra-special query
    let paymentsQuery  = Payments.query()
    paymentsQuery?.limit = limit
    paymentsQuery?.skip  = limit*loopCount
    paymentsQuery?.findObjectsInBackground(block: { (objects, error) in
        if let objects = objects {
            //print(#file.getClass(),"  ",#function,"  loopcount: ",loopCount,"      #ReturnedObjects: ",  objects.count)
            if objects.count > 0 {
                //print(#function, " no. of objects :", objects.count)
                for paymentsObject in objects {
                    let user   = paymentsObject[Utils.name] as! String
                    let amount = paymentsObject[Utils.amount] as! String
                    self.users.append(user)
                    self.payments.append(amount)
                }
                //recurse our loop with increment because we are not done
                self.loadAllPaymentDetails(loopCount + 1); //<--recurse
            }else {
                //our query has run out of steam, this else{} will be called one time only
                //if the Table had been initially empty, lets inform the user:
                if self.users.count == 1 {
                    Utils.createAlert(self, title: "No Payment has been made yet", message: "Please Encourage Users to make some Payments", buttonTitle: "Ok")
                }else {
                    self.tableView.reloadData()
                }
            }
        }else if error != nil {
            print(error!)
        }else {
            print("Unknown Error")
        }
    })
}

adapted from @deLux_247's example above.

Insectile

Answer 9

You could achieve this using CloudCode... Make a custom function you can call that will enumerate the entire collection and build a response from that but a wiser choice would be to paginate your requests, and fetch the records 1000 (or even less) at a time, adding them into your list dynamically as required.

Binni

an example would have been helfpful. – Insectile

Answer 10

GENERIC VERSION For SWIFT 4:

Warning: this is not tested!

An attempt to adapt nyxee's answer to be usable for any query:

func getAllRecords(for query: PFQuery<PFObject>, then doThis: @escaping (_ objects: [PFObject]?, _ error: Error?)->Void) {
    let limit = 1000
    var objectArray : [PFObject] = []
    query.limit = limit
    func recursiveQuery(_ loopCount:  Int = 0){
        query.skip  = limit * loopCount
        query.findObjectsInBackground(block: { (objects, error) in
            if let objects = objects {
                objectArray.append(contentsOf: objects)
                if objects.count == limit {
                    recursiveQuery(loopCount + 1)
                } else {
                    doThis(objectArray, error)
                }
            } else {
                doThis(objects, error)
            }
        })
    }
    recursiveQuery()
}

Bisulcate

Answer 11

Here's my solution for C# .NET

 List<ParseObject> allObjects = new List<ParseObject>();
 ParseQuery<ParseObject> query1 = ParseObject.GetQuery("Class");
 int totalctr = await query1.CountAsync()
 for (int i = 0; i <= totalctr / 1000; i++)
 {
     ParseQuery<ParseObject> query2 = ParseObject.GetQuery("Class").Skip(i * 1000).Limit(1000);
     IEnumerable<ParseObject> ibatch = await query2.FindAsync();
     allObjects.AddRange(ibatch);
 }

Bobine