avoiding XDMP-EXPNTREECACHEFULL and loading document

Question

I am using marklogic 4 and I have some 15000 documents (each of around 10 KB). I want to load the entire content as a document ( and convert the total documents to a single csv file and output to HTTP output stream for downloading). While I load the documents this way:

let $uri := cts:uri-match('products/documents/*.xml')
let $doc := fn:doc ($uri)

The xpath has some 15000 xmls. So fn:doc throws an error XDMP-EXPNTREECACHEFULL.

Is there any workaround for this? I cannot increase tree cache size in admin console because the number of xml files in products/documents/*.xml may increase.

Thanks.

Aboulia

Answer 1

When you want to export large quantities of XML from MarkLogic, the best technique is to write the query so that results can stream, avoiding the expanded tree cache entirely. It is a very different style of coding, though: you'll have to avoid strong typing of any kind, and refactor your code to remove FLWOR expressions. You won't be able to test any of the code in cq or qconsole, either.

Take a look at http://blakeley.com/blogofile/2012/03/19/let-free-style-and-streaming/ for some tips on how to get there. At a minimum the code sample you posted would have to become:

doc(cts:uri-match('products/documents/*.xml'))

In passing I would try to rework that to avoid the *.xml part, because it will be slower than needed. Maybe something like this?

cts:search(
  collection(),
  cts:directory-query('products/documents/', 'infinity'))

If you need to test for something more than the directory, you could add a cts:and-query with some cts:element-query test.

Pronty

Answer 2

For general information about this error, see the MarkLogic knowledge base article on XDMP-EXPNTREECACHEFULL

Williamwilliams