How receive over 1000 documents from ArangoDB collection?

Question

I have a collection. My collection has 30 000 000 documents. I want to receive all its by means AQL query. I do so:

FOR c FROM MyCollection SORT c.value ASC LIMIT 30000000 RETURN c.id

But I receives no more 1001 documents. Why?

Footbridge

Which driver / API are you using? The cursor interface uses batches of 1000 to transfer to the client. – Parotitis

I'm using standard web interface, which allows to execute AQL queries. How i can receive all document in this case? – Footbridge

Answer 1

The AQL editor in the web interface has a select box at the bottom that allows specifying the maximum number of documents to receive. The selected value defaults to 1000, so only 1000 documents will be fetched.

You can increase the number up to 5000 results using the select box. This value is an arbitrary limit, however, it protects you from actions such as trying to fetch 30M documents into your browser, which is an operation that will not work (your browser will likely run out of memory or it will come to a grinding halt when trying to render the HTML for 30M rows).

Dowell

Does it mean that if I do not set the maximum number to be over 1000, a cursor will never give me more than 1000 no matter what? I thought the cursor would divide the result into multiple batch, each have limit of 1000 documents? – Osteoplastic

Cursors have a batch size of 1000 by default. If a query produces more results, the cursor can fetch them all using multiple batches. This is also how most of our drivers operate. However, the web interface in 3.1 and before seems to fetch all query results using a single batch, and uses the user-specified result limit as the batch size. This has changed in 3.2: the web interface will now fetch results in batches of 1000. We are still limiting the number of results in the web interface by default to protect the user's browser from exploding when fetching and displaying huge query results. – Dowell

I suggested to a colleague of mine to store the result limit (e.g. "1000 results" or "all results") in local storage so the selection is remembered automatically and does not need to be adjusted all the time. I think this can be implemented for 3.2beta2. – Dowell

Just a side note, your guys have amazing communication with the community. It was one of the reason we choose Arangodb over others. – Osteoplastic

@Dowell - I am getting only 1000 document. I am using github.com/solher/arangolite. It will be great if you can help . – Adjudication

@user2449952: looking at https://github.com/solher/arangolite/blob/master/requests/aql.go, it seems that this driver does not support the full cursor API of ArangoDB. A cursor is normally created using HTTP POST to /_api/cursors, and then the results are fetched in batches. The default batch size is 1000. The driver will only fetch the first batch but not any of the remaining batches. It seems that one can set a BatchSize property somehow to make all the results go into the first batch. – Dowell

@Dowell hmm.. As a turnaround solution I wrapped the response of n document inside a dummy temp object. Working ok now. Thanks for the quick response. I have few performance doubts regarding Arango against Neo4j. Can I mail or connect with you in some handle ? – Adjudication

@user2449952: please use our community Slack channel at slack.arangodb.com or file an issue in our Github issue tracker – Dowell