Docker fails to pull the image from within Azure App Service

Question

The Container Setting on the App Service it self look solid:

But the log pane shows errors:

2020-02-11 06:31:40.621 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 06:31:41.240 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
2020-02-11 06:36:05.546 INFO  - Starting container for site
2020-02-11 06:36:05.551 INFO  - docker run -d -p 9621:8081 --name app505-dfpg-qa2-web-eastus2-gateway-apsvc_0_a9c8277e_msiProxy -e WEBSITE_SITE_NAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc.azurewebsites.net -e WEBSITE_INSTANCE_ID=7d18d5957d129d3dc3a25d7a2c85147ef57f1a6b93910c50eb850417ab59dc56 appsvc/msitokenservice:1904260237  

2020-02-11 06:36:05.552 INFO  - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-02-11 06:36:17.766 INFO  - Pulling image: a...cr/gateway:1.0.20042.2
2020-02-11 06:36:17.922 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}

2020-02-11 06:36:17.923 ERROR - Pulling docker image a...cr/gateway:1.0.20042.2 failed:
2020-02-11 06:36:17.923 INFO  - Pulling image from Docker hub: a...cr/gateway:1.0.20042.2
2020-02-11 06:36:18.092 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}

2020-02-11 06:36:18.094 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 06:36:19.062 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.

The Service Principal used to deploy the App Service has AcrPush access to the parent resource group of the container registry:

The setting are present:

I did az login with that service principal and then tried az acr login to the registry. It works fine. So what am I missing here?

EDIT 1

I know the credentials are correct, because I tested them like this:

Where I just copied the values from the app service configuration and pasted on the console. docker has no problem logging in.

It must be something else.

EDIT 2

However, I also get this:

C:\Dayforce\fintech [shelve/terraform ≡]> docker pull a...r/gateway
Using default tag: latest
Error response from daemon: pull access denied for a...r/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

So, I can login, but not pull. Very strange, because the account is configured to have AcrPush access to the container, which includes AcrPull:

EDIT 3

I was able to pull successfully when using the FQDN for the registry:

I updated the pipeline, but I still get the same errors:

2020-02-11 16:03:50.227 ERROR - Pulling docker image a...r.azurecr.io/gateway:1.0.20042.2 failed:
2020-02-11 16:03:50.228 INFO  - Pulling image from Docker hub: a...r.azurecr.io/gateway:1.0.20042.2
2020-02-11 16:03:50.266 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://a...r.azurecr.io/v2/gateway/manifests/1.0.20042.2: unauthorized: authentication required"}

2020-02-11 16:03:50.269 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 16:03:50.853 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.

EDIT 4

The only way that I found working was to enable the Admin User on the ACR and pass its credentials in the DOCKER_... variables instead of credentials of the Service Principal.

This is frustrating, I know the Service Principal can login and pull when ran locally, it is a mystery why it does not work for docker running on an App Service Host. We have another team here which faced the same issue and they have not found any solution, but enable the Admin User.

EDIT 5

The entire process runs as part of the Azure DevOps on-prem release pipeline using a dedicated Service Principal. Let me call it Pod Deploy Service Principal or just SP for short.

Let DOCKER_xyz denote the three app settings controlling the docker running on the App Service host:

• DOCKER_REGISTRY_SERVER_URL
• DOCKER_REGISTRY_SERVER_USERNAME
• DOCKER_REGISTRY_SERVER_PASSWORD

I think we need to distinguish two parts here:

1. App Service needs to talk to the ACR in order to pull from it the details about the image and present them in this GUI - For that to work, the SP must have the AcrPull role in the ACR. Failure to do so results in the GUI presenting a spinning icon for the Image and Tag rows. I stumbled on it before - How to configure an Azure app service to pull images from an ACR with terraform? Now the answer to that question suggests that I have to assign the AcrPull role and set the DOCKER_xyz app settings. I think that the DOCKER_xyz app settings are not for that, but for the second part.
2. It seems to me that when an App Service is started, the host uses docker to actually pull the right image from the ACR. This part seems to be detached from (1). For it to work, the app settings must have the DOCKER_xyz app settings.

My problem is that part (1) works great, but part (2) does not even if DOCKER_xyz app settings specify the credentials of the SP from part (1). The only way I could make it work if I point DOCKER_xyz at the Admin User of the ACR.

But that why on Earth the DOCKER_xyz app settings cannot point to the pipeline SP, which was good enough for the part (1)?

EDIT 6

The current state of affairs is this. Azure App Service is unable to communicate with an ACR except using ACR admin user and password. So, even if the docker runtime running on the App Service host machine may know how to login using any service principal, the App Service would not use any identity or Service Principal to read metadata from the ACR - only admin user and password. The relevant references are:

• https://feedback.azure.com/forums/169385-web-apps/suggestions/36145444-web-app-for-containers-acr-access-requires-admin#%7btoggle_previous_statuses%7d
• https://github.com/MicrosoftDocs/azure-docs/issues/49186

On a personal note I find it amazing that Microsoft recommends not to use ACR admin user, yet a very core piece of their offering, namely Azure App Service, depends on it being enable. Makes me wonder whether different teams in Microsoft are aware of what others are doing or not doing...

Hortatory

It seems your credential is not right. You can try to assign permission directly to the ACR, not the group. And do you enable the admin user? – Dougall

How do you deploy the image in ACR? In portal? Or use the Azure CLI with the YAML file or template? – Dougall

Please, see EDIT 4. That should prove that the image is OK. – Hortatory

What about your test? Can you share the result? – Dougall

@CharlesXu - I did not have time yet. – Hortatory

Do you still not have time to test? I receive nothing for serial days. – Dougall

No, it is on the table and will not go anywhere. But I need to clear first higher priority items. I will get there, no worries. – Hortatory

So I had this as well, I checked and the credentials Azure automatically put into my app configuration were correct... could NOT get it to work. Changed to a different password (primary to secondary) and it worked fine! All I can think of was that the first password had a + in it... as the error was about input format, and I get similar things in zsh when I have stuff like + characters. – Irriguous

My app has been working fine for the last few months, but this morning I woke up with this problem, in Production. I have no idea how to proceed. – Concentric

Answer 1

App service started pulling after doing these steps for me. :D

1. Enable Admin Access in Azure Container Registry
2. In the App service configuration, provide container registry admin credentials DOCKER_REGISTRY_SERVER_PASSWORD(admin enabled password), DOCKER_REGISTRY_SERVER_USERNAME(crxxxxxx), DOCKER_REGISTRY_SERVER_URL (https://crxxxxxx.azurecr.io)
3. Go to your app service and select identity section on the left, and click on system assigned - change status to On.
4. Now go to IAM Control container registry, add ACR pull role to App Service system assigned identity enabled on step 3.
5. Restart your App Service and wait .Changes will take few minutes to reflect so refresh your logs. (10 minutes or more)

Good luck :)

Musser

This worked very well for me, thanks @Aswin – Jonellejones

just setting the DOCKER_... variables worked for me. Thanks – Scarface

Work for me after configure docker ACR credentials to app service. Thanks – Thiourea

It worked for me, I don't know if I did something wrong or it was a bug on the Azure side to not fill those envs. – Belkisbelknap

How do you do step number 4? I can't find that – Concentric

@TomBom I also struggled (but wine was involved). Go to your registry first and then add IAM role Acr Pull and next page select radio button Managed Identity – Nachison

For this to work for me, I had also to downgrade the docker node version from 18 to 16 and add the node version to my package.json file. – Wayne

Answer 2

After a lot of research I figured out a way to resolve this without enabling Admin user

1. Create an app registration using Azure Active Directory and store the secret somewhere.
2. Go to the Azure container registry and add role assignment to this newly created app with permissions of AcrPush (which also contains AcrPull).

3. In the App service configuration, replace the variables .

DOCKER_REGISTRY_SERVER_PASSWORD with Client Secret of app registration which was saved in the first step
DOCKER_REGISTRY_SERVER_USERNAME with client Id of App registration

This should solve the Docker Api exception.

It's baffling that this is not mentioned in any Azure Container Registry documentation. Although I think it is mentioned somewhere in AAD documentation indirectly 😐.

Inhabitancy

Unfortunately, I am out of context now and will not be revisiting the infrastructure for some time. I hope others can validate your findings and upvote accordingly. – Hortatory

Can verify, this works. (Although I set up with only AcrPull access). Can't find it documented anywhere. Depending on how the deployment happens, you might have to add DOCKER_REGISTRY_SERVER_URL too. – Penultimate

Just to clarify, in #2, I should add AcrPush role to the Service Principal (subscription) created in #1? I'm not sure if I understand "App Registration" correctly. – Killie

Answer 3

From the message I got of the talk, let me solve your puzzle about the error.

I guess you deploy the image in ACR to the Web App through the Azure portal. When you use the Azure portal to deploy the Web App from the ACR, it only lets you select the ACR and image and tag, but do not let you set the credential. In this way, Azure will set it itself with the admin user and password if you enable the admin user. If you do not enable it, the error you got happens.

And if you want to use the service principal, I recommend you use the other tools, such as Azure CLI. Then you can set the docker registry credential yourself with the command az webapp config container set.

Here is the example and it works fine on my side:

With the Azure CLI, you can follow the steps here.

Update:

Here are the screenshots of the test on my side:

Dougall

My problem is different. Please, see EDIT 5 – Hortatory

I clarified in EDIT 5 - it denotes the three app settings controlling the docker running on the App Service host: DOCKER_REGISTRY_SERVER_URL, DOCKER_REGISTRY_SERVER_USERNAME, DOCKER_REGISTRY_SERVER_PASSWORD – Hortatory

@Hortatory When you say "Failure to do so results in the GUI presenting a spinning icon for the Image and Tag rows", where do you see it? Finish the deployment of the web app and in the container set? – Dougall

There is an image in #59914897 - In the container settings after the deployment of the App Service is done – Hortatory

@Hortatory The issue just shows you do not create the service principal to pull the image. If you did, you can see the images in the ACR when the deployment of the web app is finished. – Dougall

There was service principal alright, but it did not have the AcrPull role in the ACR. That is why there was the spinning icon. Once AcrPull role was assigned to that service principal, the spinning disappeared and the picture above showed up clearly indicating the image and the tag. That is what I call part (1). – Hortatory

@Hortatory The service principal with the permission AcrPull is necessary for both part (1) and (2) if you use ACR. It also needs to SP to pull the images. I think if it works for part (1), then it will also work for part (2). – Dougall

I totally agree with you. That SP never went away, but your assumption is wrong - it does not work for the part (2) and that is the gist of my SO question. I banged my head on this brick wall for hours before giving up and using ACR Admin User credentials in the part (2). The SP creds just do not work. I welcome you to try for yourself. – Hortatory

@Hortatory Yeah, I tried myself and it works perfectly. See the update. – Dougall

Hmm, interesting. Neither my team nor another team we able to make it work with the service principal. – Hortatory

Let us continue this discussion in chat. – Dougall

Long time, but I am back - please see EDIT 6. – Hortatory

@Hortatory this is an amazing job from Microsoft (not). Thanks for clarifying, I have exactly the same problem and your information helped. – Stayathome

Solved setting these environment variables! Thanks a lot – Giacobo

Answer 4

Found the answer by setting "acrUseManagedIdentityCreds" to True. The second command in this comment: https://mcmap.net/q/372709/-how-to-authenticate-with-azure-acr-from-azure-container-app-service

Edit 1: Adding the command

Here is the command that you will need to run to make this change.

az resource update --ids /subscriptions/<subscription-id>/resourceGroups/myResourceGroup/providers/Microsoft.Web/sites/<app-name>/config/web --set properties.acrUseManagedIdentityCreds=True

Hoppe

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Genny

Answer 5

Just to add to mark's amazing job of working it all through and for the fast readers: for everything to work, one of course also has to enable the admin user (who by default is disabled). For example by issuing:

az acr update -n <your-azureregistry-name> --admin-enabled true

on the console.

Koloski

we can achieve this without enabling admin user as well, refer to the answer i have posted – Inhabitancy

Answer 6

I was trying to do the same from Azure DevOps pipelines and got the same problem. I didn't find out how to make it work using the ACR name, but it works if you use your_acr_name.azurecr.io instead. If you go to the Access Keys page of your ACR you will find two values Registry name: MyCoolRegistry (doesn't work if you use this one) Login server: mycoolregistry.azurecr.io The login server is working - just put it as the containerRegistry in your Pipeline without creating a service connection.

Just in case someone is struggling with that one.

Kashmir

Answer 7

I experienced this same issue when trying to deploy an Docker application to Azure Web Apps for containers.

When I deployed the application I will get the error below when I checked the App Services > My App Service > Deployment Center > Container Logs:

DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a..my-repo/image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}.

Here's how I solved it:

The issue was that I was not specifying the full path to the image. I was supposed to include my-registry-url in the docker image-name. That is instead of just image-name I was supposed to use my-registry-url/image-name, since I am trying to pull from a private repository.

So say these are variables:

• docker image name is promiseapp
• docker-registry_url is promisecicdregistry.azurecr.io
• resource-group is dockerprojects
• app-service-plan is dockerlinuxprojects
• azure-web-app name is promiseapptest
• docker-registry-user is test-user
• docker-registry-password is 12345678

CLI:

My command will be:

az webapp create --resource-group dockerprojects --plan dockerlinuxprojects --name promiseapptest --deployment-container-image-name promisecicdregistry.azurecr.io/promiseapp

az webapp config container set --resource-group dockerprojects --name promiseapptest --docker-custom-image-name promisecicdregistry.azurecr.io/promiseapp --docker-registry-server-url https://promisecicdregistry.azurecr.io --docker-registry-server-user test-user --docker-registry-server-password 12345678

Azure Portal:

To do this on the Azure Portal, simply go to Azure App Services, select the App Service, go to Deployment Center, click on Settings and set up the Deployment pipeline.

Mekong

Answer 8

In my case, I fixed the error by using the fully qualified Azure Container Registery name like this:

xwezi.azurecr.io

The previous value was

xwezi

When I deploy manually to App Services, I wouldn't get that error. But, when I used Azure App Service deploy task to deploy the container to the App Service, the service won't work correctly. And, the log stream will show the above errors.

Unfortunately, the error messages weren't helpful for me to find this out. But I hope this will save your time :)

Admit

Answer 9

I had same issue, but in my case I had different tag name in my Build pipeline and Release Pipeline.

Tallyho

Answer 10

Using only application_stack and add ignore_changes worked for me.

resource "azurerm_linux_web_app" "app" {
  name                       = "app-${var.PRODUCT}"
  

  site_config {
    
    application_stack {
      docker_image_name        = "my-image:e3920202"
      docker_registry_url      = "https://${data.azurerm_container_registry.acr.login_server}"
      docker_registry_username = data.azurerm_container_registry.acr.admin_username
      docker_registry_password = data.azurerm_container_registry.acr.admin_password
    }


  lifecycle {
    ignore_changes = [
      app_settings["DOCKER_CUSTOM_IMAGE_NAME"],
      site_config[0].application_stack[0].docker_image_name
    ]
  }
}

reference: https://github.com/hashicorp/terraform-provider-azurerm/issues/16585#issuecomment-1675045540

Reflexion