Unable to get access to Key Vault using Azure MSI on App Service

Question

I have enabled Managed Service Identities on an App Service. However, my WebJobs seem unable to access the keys.

They report:

Tried the following 3 methods to get an access token, but none of them worked. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. password_required_for_managed_user: Password is required for managed user Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,

Kudo does not show any MSI_ environmental variables.

How is this supposed to work? This is an existing App Service Plan.

Greenwood

I wanna add that I had this issue today and after checking that everything was in order I tried just restarting the App Service. It solved the problem. – Ibbison

Answer 1

The AppAuthentication library leverages an internal endpoint in App Service that receives the tokens on your site's behalf. This endpoint is non-static and therefore is set to an environment variable. After activating MSI for your site through ARM, your site will need to be restarted to get two new Environment Variables set in it:

MSI_ENDPOINT and MSI_SECRET

The presence of these variables are essential to the MSI feature working properly during runtime as the AppAuthentication library uses them to get the authorization token. The error message reflects this:

Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.

If these variables are absent, you might need to restart the site.

https://learn.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity

If the environment variables are set and you still see the same error, the article above has a code sample showing how to send requests to that endpoint manually.

public static async Task<HttpResponseMessage> GetToken(string resource, string apiversion)  {
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Secret", Environment.GetEnvironmentVariable("MSI_SECRET"));
return await client.GetAsync(String.Format("{0}/?resource={1}&api-version={2}", Environment.GetEnvironmentVariable("MSI_ENDPOINT"), resource, apiversion));

}

I would try that and see what kind of response I get back.

Bourg

So, I tried restarting the service. Numerous times. Nothing seems to make it work. Next step is to do it by hand. But, I feel that the problem is probably just that the env variable doesn't exist. – Greenwood

I should add, everytime I deploy the site again (with an Arm template) the enable MSI is disabled. I have the arm template set up to SystemAssigned. – Greenwood

In your ARM template, make sure that the "Identity" block is not within the "Properties" block. You can go to resources.azure.com and look up your site to see it's ARM properties. If MSI is enabled, You should be able to find a TenantId and a PrincipalId within an "identity" object. In the portal, make sure the application setting WEBSITE_DISABLE_MSI is set to "false". If all of that fails, you should follow this article to allow the app service team to investigate into your case: github.com/projectkudu/kudu/wiki/… – Bourg

So, it looks like it's not surviving a slot swap. My deployment process is creating a new slot, deploying to it, then swapping with prod. – Greenwood

Ah I see. We are aware of an issue with MSI in deployment slots on app service, which might explain the issue you are hitting here and we are looking into getting it fixed. – Bourg

Well, I just turned off all that slot swap stuff. Now the policy stays applied, so that's a good start. But apps still can't load the stuff. – Greenwood

Can anyone explain more about the significance of the WEBSITE_DISABLE_MSI app setting? What exactly does it do and when? I’m using terraform to set up the app service and MSI so I would need to this app setting in the tf file for terraform not to delete it. – Stupor

@JamesChristianson Is there any update on this? I'm experiencing the same issue: App Service with Deployment Slots and suddenly after a new code push the Prod slot it's unable to start, even though in staging it's working fine. – Rinehart

Answer 2

I just solved this issue when trying to use MSI with a Function app, though I already had the environment variables set. I tried restarting multiple times to no success. What I ended up doing was manually turning off MSI for the Function, then re-enabling it. This wasn't ideal, but it worked.

Hope it helps!

Lafontaine

Answer 3

I've found out that if you enable MSI and then swap out the slot, the functionality leaves with the slot change. You can re-enable it by switching it off and on again but that will create a new identity in AD and will require you to reset permissions on the key vault for it to work.

Fur

Answer 4

Enable the identity and give access to your azure function app in keyvault via access policy. You can find identity in platform feature tab These two steps works for me

Privilege

Answer 5

In my case I had forgotten to add an Access Policy for the application in the Key Vault

Inattentive

Answer 6

For the ones, like my self, wondering how to enable MSI.

My scenario: I have an App Service already deployed and running for a long time. In addition, on Azure DevOps I have my Pipeline configured to Auto-Swap my Deployment Slots (Staging/Production). Suddenly, after a normal push, Production starts failing because of the described issue.

So, in order to enable MSI again (I don't know why it has to be re-enabled but I believe this is only a workaround, not a solution, as it should be still enabled in the first place)

Go to your App Service. Then Under Settings --> Identity. Check the status: In my case, it was off

I have attached an image below to make it easier to follow.

Rinehart

Answer 7

Just switched ON the Status like @Sebastian Inones showed. Than add access policy for KeyVault like

This is resolved the issue!!

Obscurity

I would be grateful then if you can vote up my answer. – Rinehart

Answer 8

For the folks that will come across these answers, I would like to share my experience.

I got this problem with Azure Synapse pipeline run. Essentially I added access policies properly to the KeyVault, and also I added a LinkedService to the Azure Synapse pointing to my KeyVault.

If I trigger the notebook manually it works, but in the pipeline, it fails.

Initially, I used the following statement:

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString")

Then I added the name of the linked service as a third parameter, and the pipeline was able to leverage that linked service to obtain the MSI token for a Vault.

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")

Octant

Answer 9

Go to Tools->Options->Azure Services Authentication, and re-authenticate the account you want to use.

Lungi

Answer 10

Might be unrelated to your issue but I was getting the same error message.

For me, the issue was using pip3's azure-cli. I was able to fix this issue by using brew packages for both azure-cli and azure-functions-core-tools.

Uninstall pip3 azure-cli

pip3 uninstall azure-cli

Install brew azure-cli

brew update
brew install azure-cli

Kashakashden