Where is Azure Event Hub messages stored?

Question

I generated a SAS signature using this RedDog tool and successfully sent a message to Event Hub using the Events Hub API refs. I know it was successful because I got a 201 Created response from the endpoint.

This tiny success brought about a question that I have not been able to find an answer to:

I went to the azure portal and could not see the messages I created anywhere. Further reading revealed that I needed to create a storage account; I stumbled on some C# examples (EventProcessorHost) which requires the storage account creds etc.

Question is, are there any APIs I can use to persist the data? I do not want to use the C# tool.

Please correct me if my approach is wrong, but my aim is to be able to post telemetries to EventHub, persist the data and perform some analytics operations on it. The telemetry data should be viewable on Azure.

Gravante

Answer 1

You don't have direct access to the transient storage used for EventHub messages, but you could write a consumer that reads from the EventHub continuously and persist the messages to Azure Table or to Azure Blob.

The closest thing you will find to a way to automatically persist messages (as with Amazon Kinesis Firehose vs Amazon Kinesis which EventHubs are basically equivalent to), would be to use Azure Streaming Analytics configured to write the output either to Azure Blob or to Azure Table. This example shows how to set up a Streaming Analytics job that passes the data through and stores it in SQL, but you can see the UI where you can choose a choice such as Azure Table. Or you can get an idea of the options from the output API.

Of course you should be aware of the requirements around serialization that led to this question

Solidstate

Nice, thank you. That was really helpful! I followed the instructions in the example and got it working to suit my use case. One last question though, what would the EventProcessorHost do apart from what I have just done - send events, persist them, read persisted values from the db into my app? The src code isn't available so I can't do much digging. – Gravante

EventProcessorHost is a tool to take some of the pain away from handling consuming multiple partitions across different boxes. What "consumption" means varies with use case. We have a consumer that saves events to AzureTable, and a consumer (of the same data) that calculates various metrics and takes a variety of other actions. Basically it lets other C# code access the messages and do whatever. – Solidstate

With a Standard tier Event Hub, you can now have Azure automatically forward your EventHub messages to Azure Storage or Azure Data Lake. They call it Capturing. [Info is here.][1] Unlike a Streaming Analytics job, there is no cost associated with the processing of the Captured messages. You, of course, have to pay the storage costs. And there is no mechanism to cleaning up those messages over time once you've Captured them. [1]: learn.microsoft.com/en-us/azure/event-hubs/… – Ernaernald

Answer 2

The Event Hub stores data for maximum of 7 days; that’s too in standard pricing tier. If you want to persist the data for longer in a storage account, you can use the Event Hub Capture feature. You don’t have to write a single line of code to achieve this. You can configure it through Portal or ARM template. This is described in this document - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview

The event hub stores it’s transient data in Azure storage. It doesn’t give any more detail in relation to the data storage. This is evident from this documentation - https://learn.microsoft.com/en-us/azure/event-hubs/configure-customer-managed-key

The storage account you need for EventProcessorHost is only used for checkpointing or maintaining the offset of the last read event in a partition.

Brezin