"Unexpected Response Code for Operation: 0" when executing Azure Table Storage batch delete

Question

I'm using version 4.3.0 of the Windows Azure Storage libraries for .NET. In my ATS repository class, I have a couple of batch delete methods that look like this:

public async Task DeleteAsync(IEnumerable<T> entities)
{
    await ExecuteAsBatch(entities, (batch, entity) => batch.Delete(entity));
}

private async Task ExecuteAsBatch(IEnumerable<T> entities, Action<TableBatchOperation, T> batchAction)
{
    var byPartition = entities.GroupBy(x => x.PartitionKey).ToList();

    await byPartition.ForEachParallel(async group =>
    {
        // A maximum of 100 actions are allowed per batch job
        var segments = group.ToList().ToSegmentedList(100);
        await segments.ForEachParallel(async segment =>
        {
            var batch = new TableBatchOperation();
            foreach (var entity in segment)
            {
                batchAction(batch, entity);
            }
            await Table.ExecuteBatchAsync(batch);
        }, 10);
    }, 10);
}

In other places in my code, that DeleteAsync() method works correctly. However, in one particular place, I get this error message when executing the batch:

Unexpected Response Code for Operation: 0

Here's the call site:

private async Task MergeAtsOrganizationUserEvents(int organizationId, IEnumerable<CustomerUserEvent> fromEvents, CustomerUser to)
{
    var toDelete = (await fromEvents.SelectParallel(async fromEvent =>
    {
        var pkey = AtsOrganizationUserEventByMinute.GetPartitionKey(organizationId, fromEvent.OccurredOn);
        var rkey = AtsOrganizationUserEventByMinute.GetRowKey(fromEvent.OccurredOn, fromEvent.CustomerUserEventId);
        return await Ats.OrganizationUserEventByMinute.FindByPartitionRowAsync(pkey, rkey);
    })).Where(x => x != null).ToList();

    var toInsert = toDelete
        .Select(x => AtsOrganizationUserEventByMinute.FromBase(x.OrganizationId, x.OccurredOn, x.CookieId,
            to.CustomerUserId, x))
        .ToList();

    try
    {
        await Ats.OrganizationUserEventByMinute.UpsertAsync(toInsert);
        await Ats.OrganizationUserEventByMinute.DeleteAsync(toDelete);
    }
    catch (Exception ex)
    {
        _logger.Error("Unable to merge {0} AtsOrganizationEvents for org {1}, to customer user {2}: {3}",
            toInsert.Count, organizationId, to.CustomerUserId, ex.CompleteMessage());
        throw;
    }
}

The UpsertAsync() method above succeeds, but the DeleteAsync() fails. Note that it fails to delete precisely the same entities that FindByPartitionRowAsync() retrieved from the table, so I can't quite imagine how it could have anything to do with malformed entities or anything of that ilk.

Here's an example of one of the "toDelete" objects (in JSON format):

{  
   "CookieId":null,
   "CustomerUserId":185766,
   "CustomerUserEventId":3568687,
   "OrganizationId":4190,
   "EventName":"event1",
   "SessionId":null,
   "OccurredOn":"2014-10-20T18:17:09.9971379Z",
   "UrlId":null,
   "Url":null,
   "ReferrerUrlId":null,
   "ReferrerUrl":null,
   "IsSynthetic":false,
   "IpAddress":null,
   "PartitionKey":"4190.2014.10.20",
   "RowKey":"18.17.3568687",
   "Timestamp":"2014-10-20T18:17:11.237+00:00",
   "ETag":"W/\\"   datetime'2014-10-20T18%3A17%3A11.237Z'\\""
}

Azure Storage error messages are notoriously and spectacularly unhelpful, and Googling has returned nothing about batch deletes failing with this particular error.

This fails both when using local development storage and in production.

Any thoughts?

Acacia

Hi Ken, would you mind sharing your .ForEachParallel() and .ToSegmentedList() code it's something that looks very handy. Thx. – Adorl

Sure. See gist.github.com/smithkl42/ca5b98e909952eddef1e3a076c0dd91d and gist.github.com/smithkl42/36ca70a63fe1a3bbf2d03a4d0bbe46c2. – Acacia

Answer 1

'Unexpected Response Code for Operation: 0' basically means that the first operation in the batch failed. The index of the failed operation is returned in the error thrown so it makes it easier for users to go and change the specific operation in the batch that failed.

You can get more information about the request that failed and the error by catching the StorageException and checking:

• exception.RequestInformation.HttpStatusCode
• exception.RequestInformation.ExtendedErrorInformation.ErrorCode
• exception.RequestInformation.ExtendedErrorInformation.ErrorMessage

The same information is also available in the OperationContext's last result if you use an OperationContext to track the request and use suitable method overloads that take in the OperationContext.

We will look at changing the error message in the future so it is less confusing. Thanks for the feedback!

Fellner

Thanks, catching the specific StorageException gives us access to the ErrorCode and ErrorMessage which in my case explained what the problem was. Apparently, my data does not contain unique keys. "400: 22:The batch request contains multiple changes with same row key. An entity can appear only once in a batch request." – Quash

Answer 2

Another cause of this is the use of invalid characters in the key fields. If you are googling this error message you might miss this answer:

Azure Table Storage RowKey restricted Character Patterns?

Characters Disallowed in Key Fields The following characters are not allowed in values for the PartitionKey and RowKey properties:

The forward slash (/) character

The backslash () character

The number sign (#) character

The question mark (?) character

Control characters from U+0000 to U+001F, including:

The horizontal tab (\t) character

The linefeed (\n) character

The carriage return (\r) character

Control characters from U+007F to U+009F

Sapotaceous

Answer 3

in my case it got resolved for the error. 'Microsoft.WindowsAzure.Storage.StorageException: 'Element 0 in the batch returned an unexpected response code'

code snippet

table.CreateIfNotExists();

Main Code

CloudStorageAccount SA = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("SC"));

CloudTableClient tableClient = SA.CreateCloudTableClient();

CloudTable table = tableClient.GetTableReference("myWorld");

table.CreateIfNotExists();

TableBatchOperation batchOperation = new TableBatchOperation();

batchOperation.Insert(object);

table.ExecuteBatch(batchOperation);

Antionetteantioxidant