Azure Batch Pool: How do I use a custom VM Image via Python?

Question

I want to create my Pool using Python. I can do this when using an image (Ubuntu Server 16.04) from the marketplace, but I want to use a custom image (but also Ubuntu Server 16.04) -- one which I have prepared with the desired libraries and setup.

This is how I am creating my pool:

new_pool = batch.models.PoolAddParameter(
      id=pool_id,
      virtual_machine_configuration=batchmodels.VirtualMachineConfiguration(
          image_reference=image_ref_to_use, # ??
          node_agent_sku_id=sku_to_use),
      vm_size=_POOL_VM_SIZE,
      target_dedicated_nodes=_POOL_NODE_COUNT,
      start_task=start_task,
      max_tasks_per_node=_CORES_PER_NODE
)

I imaging that I need to use batch.models.ImageReference() to create my image reference... but I do not know how to use it.

Yes, I checked the documentation, which says the following:

A reference to an Azure Virtual Machines Marketplace image or a custom Azure Virtual Machine image.

It lists the parameters as:

• publisher (str)
• offer (str)
• sku (str)
• version (str)
• virtual_machine_image_id (str)

However, the parameter virtual_machine_image_id does not exists... In other words, batch.models.ImageReference(virtual_machine_image_id) is not allowed.

How can I use a custom image for my Pool?

UPDATE

So I figured out how to use a custom image... it turns out that no matter how many times I uninstall the azure python libraries and re-install them, the virtual_machine_image_id is never available.

I then went here downloaded the zip. Opened it up, checked the ImageReference class and low-and-behold, the virtual_machine_image_id was available in the __init__ function of the ImageReference class. I then downloaded the python wheel and used pip to install it. Boom it worked.

Or so I thought.

I then had to fight though trying to figure out what the node_agent_sku_id is... only by manually creating a Pool and seeing the Batch Node Agent SKU ID field did I manage to find it.

Now I am struggling with the Authentication...

The error I am getting is:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

AuthenticationErrorDetail: The specified type of authentication SharedKey is not allowed when external resources of type Compute are linked.

azure.batch.models.batch_error.BatchErrorException: {'lang': 'en-US', 'value': 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:f8c1a3b3-65c4-4efd-9c4f-75c5c253f992\nTime:2017-10-15T20:36:06.7898187Z'}

From the error, I understand that I am not allowed to use SharedKeyCredentials:

credentials = batchauth.SharedKeyCredentials(_BATCH_ACCOUNT_NAME,
                                             _BATCH_ACCOUNT_KEY)

batch_client = batch.BatchServiceClient(
    credentials,
    base_url=_BATCH_ACCOUNT_URL)

What must I do?

UPDATE 2

OK. User fpark has informed me that I need to use:

from azure.batch import BatchServiceClient
from azure.common.credentials import ServicePrincipalCredentials

credentials = ServicePrincipalCredentials(
    client_id=CLIENT_ID,
    secret=SECRET,
    tenant=TENANT_ID,
    resource="https://batch.core.windows.net/"
)
    batch_client = BatchServiceClient(
    credentials,
    base_url=BATCH_ACCOUNT_URL
)

to authenticate. Unfortunately, that the code above is described here and makes no reference to what CLIENT_ID et. al are.

I then managed to find another piece of documentation which appears to be the same thing: https://azure-sdk-for-python.readthedocs.io/en/v2.0.0rc3/resourcemanagementauthentication.html

That page pointed me to another webpage: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal

I followed that tutorial and managed to finally authenticate my application...

NOTE

When creating your application, the tutorial will tell you:

Provide a name and URL for the application. Select either Web app / API or Native for the type of application you want to create. After setting the values, select Create.

DO NOT select Native as you will not have the option to get an application key...

Throes

Can you post a version of your completed, working code? – Saad

This is super helpful - how did you figure out you needed to prepend /subscriptions/ before your image name? – Saad

I finally found the detailed error text that Azure returns in the HTTP logger under odata.metadata, in which they specify the proper format for the virtual_machine_image_id. This led me to realize that you need to copy the "resource id" field for the image on the Azure portal that can be seen when viewing the VM image details. – Saad

5 years on this is still relevant and useful. Saved me a lot of time. Thank you. – Anecdotic

Answer 1

Required Minimum Azure Batch SDK

The azure-batch Python SDK v4.0.0 or higher is required. Typically with pip install --upgrade azure-batch you should just get the newest version. If that doesn't work you can add the --force-reinstall option to pip to force it (with --upgrade).

Node Agent Sku Id

Regarding the proper value for node_agent_sku_id, you need to use the list_node_agent_skus operation to see the mapping between operating systems and the node agent skus supported.

Azure Active Directory Authentication Required

Regarding the auth issue, you must use Azure Active Directory authentication to use this feature. It will not work with shared key auth.

Documentation

More information can be found in this guide, including all pre-requisites needed to enable custom images.

Pastiche

Thanks, do you have an example of how to use the Azure Active Directory authentication? How can I do this? What do I pass into batch.BatchServiceClient( credentials, base_url=_BATCH_ACCOUNT_URL)? – Throes

Answer 2

I am using azure-batch==9.0.0, and it turns out the docs are not updated as per the package itself. Using id instead of virtual_machine_image_id fixes the problem for me.

Psychopharmacology