Azure IoT hub is taking more than 2 sec to push messages to mqtt device

Question

We have a number of devices (around 50-60 devices) connecting via MQTT to IoTHub. To save battery, our devices wake up once every 5 min, stay awake for 1 sec to get any messages and if there is nothing they goes back to sleep. This was working fine until about 2 weeks ago when this stopped working.

After trying a few things, we found that if we increase our Stay-Awake time to 3 sec, then when devices connect, the IoTHub pushes the messages to the devices. On average the IoT hub is taking about 2.5 sec to push messages once the devices connect. This will drain our battery very quick and we cannot use this as a solution. It seems to be too long to wait for messages to be dispatched considering that the devices are in the same region as the IoTHub.

We have also noticed that sometimes connection timeout needed to be increased to nearly 10 sec to get a successful connection, which seems to be far too long.

On our IoT devices, we are using MQTT protocol and due to limited hardware resources we could not use the Azure IoT sdk

So the questions here are:

Is it normal for the IoTHub to take 2-3 seconds to push mqtt messages once the device connect?

And is it normal to take up to 10 seconds to establish a connection with IoTHub?

Update 1 - 5th May 2017 10:11 AM

I have raised this question with Microsoft via MSDN, here is the link to the question: https://social.msdn.microsoft.com/Forums/en-US/9d24c261-2280-4515-8e89-8097625307ee/azure-iothub-is-taking-more-than-23-seconds-to-dispatch-messages-via-mqtt-to-the-devices-when?forum=azureiothub

We are using the Azure sdk on our cloud services. However on the IoT devices we could not use the sdk due to hardware limitation, so we are just using MQTT protocol to connect, subscribe, process messages, then disconnect. We are located in Australia (Melbourne SE) and we have multiple deployments that we are testing against, some are here in AU and some in East US, all deployments are giving us similar extended wait time to connect and to dispatch / push messages to the IoT devices

Anandrous

Have you tried contacting Microsoft Azure Support with this question? – Kef

Thanks @ChrisPietschmann, I have raised with Microsoft and have not heard back from them. I have raised the question here hoping to get an answer from the community – Anandrous

where have you raised the question with Microsoft? and what SDK are you using? – Oleic

@pierreca-MSFT, here is the link to the msdn question social.msdn.microsoft.com/Forums/en-US/… – Anandrous

and which SDK are you using? feel free to edit the question directly to add all relevant details; including the code that connects and send the message. – Oleic

thanks @pierreca-MSFT, I have updated the questions with more details – Anandrous

so, long answer short, not it's not normal. hard to say why though: If you were in touch with support, they would ask you a way to repro the issue, as well as the name of a hub you're seeing the problem on, and a timestamp to look around for them to look into details on the msft side. Right now there are at least half a dozen possibilities where this slow down could come from, between the mqtt stack, the rtos, the network, the cloud itself... can you share code? logs? if you'd rather discuss in private, may I suggest azure.microsoft.com/en-us/support/options ? – Oleic

How much time does your device to take for waking up before starting to connect the hub? And what kind of your device? – Community

Thanks for the comments here everyone. I have opened a support ticket with Microsoft and will update this question with an answer once I find the solution – Anandrous

Answer 1

Ok here are my findings based on the conversations with Microsoft Support and the IotHub product team. I am not sure why the question was voted down. It's a common problem that could challenge anybody building an IoT solution and I think there is a good value in sharing this.

The summary of Microsoft response is:

1. The IoTHub team had found that they had a defect that was pushed around March 2017, which has increased the latency of messages pushed to the devices by about 1 sec. This is what has caused our solution to break. This is because we were assuming that our IoT devices can get the messages from the IoTHub within 1 sec of reconnecting.

2. Microsoft has advised that we cannot count on a specific latency as it is not included in their SLA. The point here is that if you are designing an IoT solution, make sure that you do not assume any set latency as this time could fluctuate between hundreds of milliseconds and 10 sec.

Update 19th May 2017

Microsoft has just responded with more details. They have now added a section in the docs of the Azure IoTHub to say that changing or scaling the tier of your IotHub does not change the latency and to confirm that latency is not part of the SLA. Here is the details: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-quotas-throttling#latency

End of Update

I hope this would help other people that might have the same problem

Anandrous