EC2 User Data not working via python boto command

Question

I am trying to launch an instance, have a script run the first time it launches as part of userdata. The following code was used (python boto3 library):

import boto3
ec2 = boto3.resource('ec2')
instance = ec2.create_instances(DryRun=False, ImageId='ami-abcd1234', MinCount=1, MaxCount=1, KeyName='tde', Placement={'AvailabilityZone': 'us-west-2a'}, SecurityGroupIds=['sg-abcd1234'], UserData=user_data, InstanceType='c3.xlarge', SubnetId='subnet-abcd1234')

I have been playing around with the user_data and have had no success. I have been trying to echo some string to a new file in an existing directory. Below is the latest version I attempted.

user_data = '''
    #!/bin/bash
    echo 'test' > /home/ec2/test.txt
    '''

The ami is a CentOS based private AMI. I have tested the commands locally on the server and gotten them to work. But when I put the same command on the userdata (tweaked slightly to match the userdata format), it does not work. Instance launches successfully but the file I specified is not present.

I looked at other examples (https://github.com/coresoftwaregroup/boto-examples/blob/master/32-create-instance-enhanced-with-user-data.py) and even copied their commands.

Your help is appreciated! Thanks :)

Frady

Answer 1

For User Data to be recognized as a script, the very first characters MUST be #! (at least for Linux instances).

However, your user_data variable is being defined as:

"\n    #!/bin/bash\n    echo 'test' > /home/ec2/test.txt\n    "

You should define it like this:

user_data = '''#!/bin/bash
echo 'test' > /tmp/hello'''

Which produces:

"#!/bin/bash\necho 'test' > /tmp/hello"

That works correctly.

So, here's the final product:

import boto3

ec2 = boto3.resource('ec2')

user_data = '''#!/bin/bash
echo 'test' > /tmp/hello'''

instance = ec2.create_instances(ImageId='ami-abcd1234', MinCount=1, MaxCount=1, KeyName='my-key', SecurityGroupIds=['sg-abcd1234'], UserData=user_data, InstanceType='t2.nano', SubnetId='subnet-abcd1234')

After logging in:

[ec2-user@ip-172-31-2-151 ~]$ ls /tmp
hello  hsperfdata_root

Weinreb

Still did not work - I updated my user_data to match yours: >>> user_data = '''#!/bin/bash ... echo 'test' > /tmp/hello''' I launched a server and logged in: [root@ip-172-31-26-28 ~]# ll -tr /tmp/ total 8 drwxr-xr-x 2 elasticsearch elasticsearch 4096 Feb 16 2016 elasticsearch drwxr-xr-x 2 elasticsearch elasticsearch 4096 Feb 17 2016 jna--1985354563 [root@ip-172-31-26-28 ~]# Cheers – Frady

it did not solve my question. If you read my comment to your first answer, you will see I tried your suggestion and it still did not work. Any thoughts? – Frady

Does User Data work for that AMI if you launch it via the EC2 Management Console and manually enter the User Data? The AMI might not be configured for cloud-init. – Weinreb

Thanks John! Appreciate the follow up. I see the user data when I curl 169.254.169.254/latest/user-data. Unfortunately it did not execute it (i re-ran the same command as you suggested in your first answer). Any useful links on how to configure an instance for cloud-init + then save it as an AMI image? If that is the way to go. Cheers! – Frady

I've added a full example above, with output showing that it works. If it doesn't work for you, then the problem lies with the AMI not being configured for cloud-init. (All AWS Linux AMIs are configured with cloud-init by default.) I'm not sure what you mean by "configure an instance for cloud-init + then save it as an AMI image" -- User Data is not kept within an AMI, it is specified when the instance is launched. Alternatively, you could configure Linux to run a command at startup, without involving User Data. (Search elsewhere to find out how!) – Weinreb

Thanks @John Rotenstein. Appreciate your help. Looks like this AMI image is not configured for cloud-init. I'll use the user data to simply pass data as needed (if I use it at all). – Frady

How to define if we use userdata with windows EC2 instance? – Oversew

@LakshminarayananS See: Running commands on your Windows instance at launch - Amazon Elastic Compute Cloud – Weinreb

@JohnRotenstein, I just followed one document to use userdata for windows instance using Boto3. Instance are launched, but user data is not applied. Shall i post my code here? – Oversew

@LakshminarayananS Please create a new question rather than asking via a comment on an old question. – Weinreb

Yeah, sure i will create new question on stack-overflow. – Oversew

Answer 2

Maybe your script did not worked because you typed the wrong user path.

The correct user path is /home/ec2-user, and not /home/ec2 as supposed in your script.

Try this as user data:

user_data = '''
    #!/bin/bash
    echo 'test' > /home/ec2-user/test.txt
    '''

Grant

Answer 3

I believe you need to specify an instance profile. Have a look at this page: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-api-cli

The steps to do this include:

• Create a role
• Attach the appropriate policy document to that role (note: AmazonEC2FullAccess policy as used in the example might not be it, I just haven't found a better alternative yet)
• Create an instance profile
• Attach the role to the instance profile
• Create instances using the instance profile

Additionally as mentioned in other answers start your user data with

#!/bin/bash

Here is a "simplified" code snippet, I'm using to do the same. This skips over other initialization steps such as picking a machine image, and setting up a security group and such.

iam_client = boto3.client('iam')

# create assume role policy document
assume_role_policy_document = json.dumps({
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
        }
    ]
})

# create ec2 full access role
role_response = iam_client.create_role(
    RoleName='ec2_full_access_role',
    AssumeRolePolicyDocument=assume_role_policy_document,
    Description='AmazonEC2FullAccess',
    MaxSessionDuration=3600,
)

# get policy arm
policy_arn = policies_df[policies_df.PolicyName=='AmazonEC2FullAccess'].Arn.iloc[0]

# attach policy
role_attach_response = iam_client.attach_role_policy(
    RoleName='ec2_full_access_role', 
    PolicyArn=policy_arn
)

# create instance profile
instance_profile_response = iam_client.create_instance_profile(
    InstanceProfileName='ec2_instance_profile',
)

# attach role to instance profile
attach_role_response = iam_client.add_role_to_instance_profile(
    InstanceProfileName='ec2_instance_profile',
    RoleName='ec2_full_access_role'
)

# define user data
user_data = """#!/bin/bash
touch test.txt
"""

# create instance
instance_params = {
    'ImageId': image_id,
    'InstanceType': instance_type,
    'KeyName': key_name,
    'SecurityGroups': (sg_name, ),
    'UserData': user_data,
    'IamInstanceProfile':{'Name': 'ec2_instance_profile'},
    'MinCount': 1, 
    'MaxCount': 1
}
instances = ec2_res.create_instances(**instance_params)

Photothermic

Answer 4

In case anyone is attempting to make user_data work with the amazon.aws.ec2_instance Ansible module, for bootstrapping EKS worker nodes:

user_data: "#!/bin/bash \n
            sudo /etc/eks/bootstrap.sh 
            --apiserver-endpoint {{ CLUSTER_ENDPOINT }}
            --b64-cluster-ca {{ CERTIFICATE_AUTHORITY_DATA }} 
            {{ CLUSTER_NAME }}"

The newline is the important part here.

Sent