DQL query to return all files in a Cabinet in Documentum?

Question

I want to retrieve all the files from a cabinet (called 'Wombat Insurance Co'). Currently I am using this DQL query:

select r_object_id, object_name from dm_document(all) 
where folder('/Wombat Insurance Co', descend);

This is ok except it only returns a maximum of 100 results. If there are 5000 files in the cabinet I want to get all 5000 results. Is there a way to use pagination to get all the results?

I have tried this query:

select r_object_id, object_name from dm_document(all) 
where folder('/Wombat Insurance Co', descend) 
ENABLE (RETURN_RANGE 0 100 'r_object_id DESC');

with the intention of getting results in 100 file increments, but this query gives me an error when I try to execute it. The error says this:

com.emc.documentum.fs.services.core.CoreServiceException: "QUERY" action failed.

java.lang.Exception: [DM_QUERY2_E_UNRECOGNIZED_HINT]error:  
"RETURN_RANGE is an unknown hint or is being used incorrectly."

I think I am using the RETURN_RANGE hint correctly, but maybe I'm not. Any help would be appreciated!

I have also tried using the hint ENABLE(FETCH_ALL_RESULTS 0) but this still only returns a maximum of 100 results.

To clarify, my question is: how can I get all the files from a cabinet?

Groveman

Answer 1

Aha, I've figured it out. Using DFS with Java (an abstraction layer on top of DFC) you can set the starting index for query results:

String queryStr = "select r_object_id, object_name from dm_document(all) 
                   where folder('/Wombat Insurance Co', descend);"

PassthroughQuery query = new PassthroughQuery();
query.setQueryString(queryStr);
query.addRepository(repositoryStr);

QueryExecution queryEx = new QueryExecution();
queryEx.setCacheStrategyType(CacheStrategyType.DEFAULT_CACHE_STRATEGY);
queryEx.setStartingIndex(currentIndex);      // set start index here

OperationOptions operationOptions = null;

// will return 100 results starting from currentIndex
QueryResult queryResult = queryService.execute(query, queryEx, operationOptions);

You can just increment the currentIndex variable to get all results.

Groveman

Answer 2

Well, the hint is being used incorrectly. Start with 1, not 0.

There is no built-in limit in DQL itself. All results are returned by default. The reason you get only 100 results must have something to do with the way you're using DFC (or whichever other client you are using). Using IDfCollection in the following way will surely return everything:

IDfQuery query = new DfQuery("SELECT r_object_id, object_name "
        + "FROM dm_document(all) WHERE FOLDER('/System', DESCEND)");
IDfCollection coll = query.execute(session, IDfQuery.DF_READ_QUERY);
int i = 0;
while (coll.next()) i++;
System.out.println("Number of results: " + i);

In a test environment (CS 6.7 SP1 x64, MS SQL), this outputs:

Number of results: 37162

Now, there's proof. Using paging is however a good idea if you want to improve the overall performance in your application. As mentioned, start counting with the number 1:

ENABLE(RETURN_RANGE 1 100 'r_object_id DESC')

This way of paging requires that sorting be specified in the hint rather than as a DQL statement. If all you want is the first 100 records, try this hint instead:

ENABLE(RETURN_TOP 100)

In this case sorting with ORDER BY will work as you'd expect.

Lastly, note that adding (all) will not only find all documents matching the specified qualification, but all versions of every document. If this was your intention, that's fine.

Hence

Answer 3

You have already accepted an answer which is using DFS.

Since your are playing with DFC, these information might help you.

DFS:

If you are using DFS, you have to aware about the number of concurrent sessions that you can consume with DFS. I think it is 100 or 150.

DFC:

Actually there is a limit that you can fetch via DFC (I'm not sure with DFS).

Go to your DFC application(webtop or da or anything) and check the dfc.properties file.

# Maximum number of results to retrieve by a query search.                      
# min value:  1, max value: 10000000
# 
dfc.search.max_results = 100

# Maximum number of results to retrieve per source by a query search.           
# min value:  1, max value: 10000000
# 
dfc.search.max_results_per_source = 400

dfc.properties.full or similar file is there and you can verify these values according to your system.

And I'm talking about the ContentServer side, not the client side dfc.properties file.

If you use ENABLE (RETURN_TOP) hint with DFC, there are 2 ways to fetch the results from the ContentServer.

1. Object based
2. Row based

You have to configure this by using the parameter return_top_results_row_based in the server.ini file.

All of these changes for the documentum server side, not for your DFC/DQL client.

Faraway

Answer 4

I've worked with DFC API (with Java) for a while but I don't remember any default limit on queries, IIRC we've always got all of the documents, there weren't any limit. Actually (according to my notes) we have to set the limit explicitly with, for example, enable (return_top 2000). (As far I know the syntax might be depend on the DBMS behind EMC Documentum.)

Just a guess: check your dfc.properties file.

Transpacific

Thanks for the suggestions. The enable (return_top 2000) still only returned 100 results and the properties file seemed to be in order. I've managed to find a solution. I'm using DFS (sorry I should have mentioned this in the question) which is an abstraction layer on top of DFC and it turns out you can set the starting index for query results. – Groveman