ExecuteQuerySegmentedAsync is not working when the Azure table has more records

Question

I'm trying to read the records from Azure table storage. I have a simple query while pulling the records

var isPagination = true;
var  combinedFilter = "groupCode eq '9THS'";
var query = new TableQuery<AzStorageEntityAdapter<T>>().Where(combinedFilter);

TableRequestOptions tableRequestOptions = new TableRequestOptions() 
                                            { ServerTimeout = TimeSpan.FromSeconds(90) };

do
{
    var segments = await table.ExecuteQuerySegmentedAsync(query, continuationToken, tableRequestOptions, null);
    currentPageResult.AddRange(segments.Results.Select(s => s.InnerObject).ToList());
    continuationToken = segments.ContinuationToken;
}  while (continuationToken != null && !isPagination);;

It was working till the azure table had less number of records(10000) with say 3 to 4 distinct Groupcodes.

When the table size increased over 200000 records, the search will not return any records (i.e) segments.Results has zero records, but the continuationToken has values.

If I replace the the ExecuteQuerySegmentedAsync with ExecuteQuery it returns the expected records. I tried to add ServerTimeout, MaximumExecutionTime nothing helped.

Whats wrong here ?

Pinter

Answer 1

There's nothing wrong here :). It is expected behavior.

From Query Timeout and Pagination:

A query against the Table service may return a maximum of 1,000 items at one time and may execute for a maximum of five seconds. If the result set contains more than 1,000 items, if the query did not complete within five seconds, or if the query crosses the partition boundary, the response includes headers which provide the developer with continuation tokens to use in order to resume the query at the next item in the result set. Continuation token headers may be returned for a Query Tables operation or a Query Entities operation.

Basically your query is doing a full table scan as you have not specified any PartitionKey in your query and it tries to search for matching records by going from top of the table to the bottom. Since it is not finding any matching entities in 5 seconds, it is simply returning the continuation token.

As to why ExecuteQuery is working is because it internally handles the continuation token. You can confirm it by tracing the request/response in Fiddler.

Aceae

So I should reiterate till I get the result ? – Meraz

Yes, you should. Other option would be to rethink about the design of your table. Ideally all of your queries must include a PartitionKey. If you think you're going to query on groupCode you may want to store a copy of the data with groupCode value as PartitionKey. – Aceae

You may find this link useful: learn.microsoft.com/en-gb/azure/cosmos-db/…. Please see Design for querying section there. – Aceae

Actually I didn't post the exact query. my while condition is like while (continuationToken != null && !isPagination); isPagination is always true so it coming out without looping. Just realized this mistake after reading your explanation. Man you are so helpful in Azure, literally life saver. – Meraz

Also, I answered a similar question that you may find useful: #15809578. – Aceae

Man you are so helpful in Azure, literally life saver. - <blush/> ... I am happy that I was able to help you out :). – Aceae

Thanks for the help. I'll read the link you have shared. I have many filters actually. – Meraz