azure servicebus maxConcurrentCalls totally ignored

Question

I have thsese in my host.json but every time i run the function it runs in parallel runing much more threads then 1 ( so much as there are messages in queue)

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "prefetchCount": 1,
      "messageHandlerOptions": {
        "maxConcurrentCalls": 1
      }
    }
  }
}

my function

 [FunctionName(nameof(ResourceEventProcessorFunction))]
    public async Task Run([ServiceBusTrigger("%TopicName%", "%SubscriptionName%", Connection = "ServiceBusConnection", IsSessionsEnabled = true)]Message message, IMessageSession messageSession, ILogger log)

Culbert

Answer 1

so the problem was that every message had a differnet sessionId. Disabling sessionId on subscription in azure solved this problem.

In details below for bounty :D azure docs doesnt exactly specify how to limit thread number, but I looked a bit dipper.

there is MessageRecievePump and SessionRecievePump one uses MaxConcurrentCalls the other one MaxConcurrentSessions and MaxConcurrentAcceptSessionCalls

be aware of this if you include session in your subscription (MaxConcurrentCalls doesnt work) it works only when session id is the same. when session is differnt try to use MaxConcurrentSessions or MaxConcurrentAcceptSessionCalls but be aware there are no docs about this....

Culbert

This only works if you have a dedicated App Service Plan with a single instance, right? For anything else, you would need to either use the Singleton attribute or use the same sessionId for all messages. See also my answer. – Bailor

Answer 2

Leveraging sessions

Since you are using sessions, you can use the same sessionId for all messages, and they will be processed in order by a single instance, regardless of the settings in your host.json.

https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions

Using Singleton attribute

If you can't use the sessionId for your purpose, you should try the [Singleton] attribute on your function. This will ensure that only one instance across all of your function instances will process the request.

We have this working successfully for WebJobs in production, and it should work just the same for Azure Functions. If you have dedicated app service plans, using this attribute should be enough. This is not recommended for a consumption plan.

[Singleton] does work on functions. The Azure Function host will create or wait for a lock in the Azure Storage account. The lock is the host ID which should be the same for all hosts of an app across all instances - so all instances share this lock and will only allow one execution to occur at a time.

To test this I put 1000 queue messages at once on a function with [Singleton]. The function would wake up, emit the invocation ID, sleep, and then emit the invocation ID. After processing all 1000 I looked at logs and never saw invocation IDs overlap. Only one invocation would happen globally at a time.

https://github.com/Azure/azure-functions-host/issues/912#issuecomment-419608830

 [Singleton]
 [FunctionName(nameof(ResourceEventProcessorFunction))]
    public async Task Run([ServiceBusTrigger("%TopicName%", "%SubscriptionName%", Connection = "ServiceBusConnection", IsSessionsEnabled = true)]Message message, IMessageSession messageSession, ILogger log)

In a consumption plan

Continuing the quote above:

With that said I think the recommendation is: [Singleton] isn't recommended for consumption hosted function plans. If you have a dedicated app service plan it's fine (as you are paying for the instance anyway). If you want to enforce [Singleton] like behavior in a consumption plan you are likely best to:

1. Set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 so you never scale to more than one instance
2. Set the host.json file to only allow 1 concurrent execution at a time for that trigger (for instance a batch size of 1 for Azure Queues).

https://github.com/Azure/azure-functions-host/issues/912#issuecomment-419608830

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "prefetchCount": 1,
      "messageHandlerOptions": {
        "maxConcurrentCalls": 1
      }
    }
  }
}

Bailor

Answer 3

Maybe you can set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 to make the function run only one instance at a time.

If you develop locally, you can set it in local.settings.json, if you develop in Azure portal, you can set it in Configuration -> Application settings.

Note：

1. If you set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1, your function will not scale out and can only run in one instance.

2. In addition to setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT, you still need to set maxConcurrentCalls to 1

3. This setting is in preview. An app property for function max scale out has been added and is the recommended way to limit scale out.

For more details, you can refer to this official document.

Taddeusz

Please try this setting. – Taddeusz

thanks Frank but it didn't worked as well. Tried this even allreday. Don't know why but I cant controll number of messages process in paralell – Culbert

Sorry for not helping you solve the problem. I encountered a problem before using [assembly:FunctionsStartup(typeof(Startup) to cause host.json to fail. You can see if it is related to this. – Taddeusz

it doesnt even recognize FunctionsStartup , but your case was eventhub , my is servicebus – Culbert

from what i understand, if you're on consumption, you do not have the ability to enforce that the function runs ONLY once because it may be running on multiple VMs. is this correct? if so, then WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT will not be affective at ensuring just 1 instance, however it WILL ensure 1 instance per node (vm) – Theran

@FrankGong please provide input on my comment above – Theran