How to fetch unlimited documents using Mango query couchDB without providing "limit" field?

Question

I am using Mango query in couch db for almost every query to find the documents. Here I am facing a problem in fetching all the documents matching the given conditions. The problem is that the default limit of mango query is 25 (means fetch 25 documents per query) and there are lot many documents in my database and I don’t have the exact count of documents. I cannot hard code the limit in mango query as I don’t know the upper limit of documents and I don’t think hard coding the limit is a good idea. Can anyone please help me with this issue? How can I make the limit as unlimited or is there any other way to handle this situation?

Gwendolin

With CouchDB 2.1, you can change the default limit. Therefore, I don't think that you can avoid this limit. Anyway, you should have a limit because the more you have results, the longer it takes to be processed and send over http. You should always design your code to handle paging of requests. – Uprear

@AlexisCôté Thanks Alexis!! :) I'll go with the pagination approach as it seems to be a better and the only option to go with at this point. – Gwendolin

My bad, the pagination is not supported yet (it looks like it will be part of the next release), I guess you can temporarly put a high number for the limit. – Uprear

Yeah, Pagination is not supported yet. Putting a high number for the limit won't work in my situation. I'll have to add custom pagination using Node JS (backend code). Any idea how to achieve that? We are using Node JS for backend and Angular JS for Frontend. I am thinking of calling Node js API from frontend for each Page request. Is it a good approach? Or is there any better idea to do this? – Gwendolin

Depending on how your database is structured, you could simulate pagination by adding another selector to your mango query to find documents were the ID is greater than the previous query. – Naamann

Answer 1

I countered the same issue and resolved it with a recursive function

const batchSize = 25;
let batchCount = 0;
let searchResults = [];

//  in my case I want to find docs with a certain date, hardcoded here as example
let selectedDate = '2017/10/30'; 

// the recursive function
let search = function (count, limit){
    return db.find({
        selector: {
            date: selectedDate  
        },
        limit: batchSize,
        skip: batchCount*batchSize 
    }).then(function(batch){
        if (batch.docs.length === 0){
            // there are no (more) search results, so we can return what we have

            return searchResults

        } else {
            // there may be more search results, so we add this batch to the result and call the function again to ask more. We increase the counter so that the first batch(es) are skipped

            for (var i = 0; i < batch.docs.length; i++) {
                searchResults.push(batch.docs[i])
            }
            batchCount ++
            return search(batchCount, batchSize) 
        }
    })
}

search(batchCount, batchSize).then(function(result){
    // you can handle the searchresults

})

Housewifely

Answer 2

Apparently there is a bookmark key that is returned on each _find call that you can then pass back to another _find call to get the next 'page' of results (assuming nothing else about the second query was altered from the first).

Not sure how practical this approach is if the previous query isn't guaranteed to be the same though.

I actually ended up using an iterative version of the @Erik's answer.

See the JSON request options in the docs here: https://docs.couchdb.org/en/latest/api/database/find.html

Chanukah