is it possible to customize the events that a blob within a storage account fires on blob creation?

Question

Is it possible to change the default event that is fired on blobcreated?

Storage accounts have the ability to fire events when blobs are deleted/created:

If you add a new event subscription, you can choose between these three:

I'd like to be able to use the Custom Input Schema. However, there's no documentation on how to use it.

How do we customize the custom input schema?

The default schema looks something like this:

{
    "topic": "/subscriptions/xxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.Storage/storageAccounts/mystoraccount",
    "subject": "/blobServices/default/containers/xmlinput/blobs/myj.json",
    "eventType": "Microsoft.Storage.BlobCreated",
    "eventTime": "2019-05-20T18:58:28.7390111Z",
    "id": "xxxxxxxxxxxxxxxx",
    "data": {
        "api": "PutBlockList",
        "clientRequestId": "xxxxxxxxxxxxxxxx",
        "requestId": "xxxxxxxxxxxxxxxx",
        "eTag": "0x8D6DD55254EBE75",
        "contentType": "application/json",  
        "contentLength": 874636,
        "blobType": "BlockBlob",
        "url": "https://mystoraccount.blob.core.windows.net/xmlinput/myj.json",
        "sequencer": "00000000000000000000000000005FAC0000000000614963",
        "storageDiagnostics": {
            "batchId": "xxxxxxxxxxxxxxxx"
        }
    },
    "dataVersion": "",
    "metadataVersion": "1"
}

I'd like to ONLY return the file name, in this case it is a substring of the subject, myj.json.

How do we customize the event that's being fired?

Desired result:

{
  "filename": "myj.json"
}

Musky

Answer 1

The Azure Event Grid supports a CustomInputSchema only for Custom and Event Domain topics. In other words, the AEG built-in event sources can be distributed only with the EventGridSchema (default schema) or CloudEventV01Schema.

For your solution, when your consumer requires to subscribe to the AEG events with a custom schema, you need to chain events to the custom topic with a CustomInputSchema. The following screen snippet shows this concept:

For topic chaining (integrator) can be used a serverless Azure Function or Api Management. In my test (like is shown in the above picture) the EventGridTrigger function has been used.

The integrator has a responsibility to fire the AEG custom topic endpoint with a custom schema.

The following code snippet shows an example of the EventGridTrigger integrator:

#r "Newtonsoft.Json"

using System;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

static HttpClient client = new HttpClient() { BaseAddress = new Uri (Environment.GetEnvironmentVariable("CustomTopicEndpointEventGrid")) };

public static async Task Run(JObject eventGridEvent, ILogger log)
{
    log.LogInformation(eventGridEvent.ToString());

    string url = $"{eventGridEvent["data"]?["url"]?.Value<string>()}";
    if(!string.IsNullOrEmpty(url))
    {
        // Fire event
        var response = await client.PostAsJsonAsync("", new[] { new { filename = url.Substring(url.LastIndexOf('/') + 1) } });
        log.LogInformation(response.ToString());
    }

    await Task.CompletedTask;
}

Note, that the CustomInputSchema is still in the preview, so to create a custom topic with a custom input schema follow docs here. Also, the REST API can be used, see more details here.

The following is my example of the payload for creating a custom topic with a CustomInputSchema using a REST Api:

    {
      "location": "westus",
      "tags": {
        "tag1": "abcd",
        "tag2": "ABCD"
      },
      "properties": {
        "inputSchema": "CustomEventSchema",
        "inputSchemaMapping": {
          "properties": {
            "id": {
              "sourceField": null
              },
            "topic": {
              "sourceField": null
              },
            "eventTime": {
              "sourceField": null
              },
            "eventType": {
              "sourceField": "myEventType",
              "defaultValue": "BlobCreated"
              },
            "subject": {
              "sourceField": "mySubject",
              "defaultValue": "/containers/xmlinput/blobs"
              },
            "dataVersion": {
              "sourceField": null,
              "defaultValue": "1.0"
              }
            },
         "inputSchemaMappingType": "Json"
        }
     }
  }

Once you have a custom topic with a CustomInputSchema, the output delivery schema will be followed by schema from the input. In the case, when your subscription on this custom topic will be delivered with an EventGridSchema, than the above mapping will be applied for the event delivery.

Sandrasandro