Apt-get with Packer randomly failing

Question

I'm using Packer to build an ami with a file ami.json that runs two provisioners built off the default Ubuntu Server 20.04 LTS image. The problem is Packer build randomly fails on apt-get install ansible with the error E: Unable to locate package ansible. The same ami.json file builds or doesn't build intermittently despite zero changes.

It seems potentially related to this question from 5 years ago that got a workaround but not a real answer: Packer/Amazon EBS/Ubuntu - Inconsistent PPAs

{
"variables" : {
    "region" : "us-west-2"
},
"builders" : [
    {
        "type" : "amazon-ebs",
        "profile" : "default",
        "region" : "{{user `region`}}",
        "instance_type" : "t2.micro",
        "source_ami" : "ami-06e54d05255faf8f6",
        "ssh_username" : "ubuntu",
        "ami_name" : "packer_build_test",
        "ami_description" : "test"
    }
],
"provisioners" : [
  {   "type" : "shell",
        "inline" : [
            "sudo add-apt-repository universe",
            "sudo apt-get update",
            "sudo apt-get install -y ansible"
        ]
    },
    {
        "type" : "ansible-local",
        "playbook_file": "./server.yml"
    }
]
}

I just ran it twice and got a failed build and then a successful one directly into a row

Output from the failed build:

    $ ~/CICDServerSetup$ packer build ami.json
amazon-ebs: output will be in this color.

==> amazon-ebs: Prevalidating any provided VPC information
==> amazon-ebs: Prevalidating AMI Name: packer_build_test_1
    amazon-ebs: Found Image ID: ami-06e54d05255faf8f6
==> amazon-ebs: Creating temporary keypair: packer_5f7a86b8-01b9-aa26-c418-bf95da74e2bd
==> amazon-ebs: Creating temporary security group for this instance: packer_5f7a86ba-5d3b-8196-770f-c2a59e3a9337
==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-0a0f119d8f4160c43
==> amazon-ebs: Waiting for instance (i-0a0f119d8f4160c43) to become ready...
==> amazon-ebs: Using ssh communicator to connect: 54.190.40.167
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell305064109
    amazon-ebs: 'universe' distribution component is already enabled for all sources.
    amazon-ebs: Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
    amazon-ebs: Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
    amazon-ebs: Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
    amazon-ebs: Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [294 kB]
    amazon-ebs: Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB]
    amazon-ebs: Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
    amazon-ebs: Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [69.3 kB]
    amazon-ebs: Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [4696 B]
    amazon-ebs: Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [59.2 kB]
    amazon-ebs: Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [9856 B]
    amazon-ebs: Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [92.3 kB]
    amazon-ebs: Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [35.0 kB]
    amazon-ebs: Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [2956 B]
    amazon-ebs: Get:14 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [1252 B]
    amazon-ebs: Get:15 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [540 B]
    amazon-ebs: Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [116 B]
    amazon-ebs: Get:17 http://archive.ubuntu.com/ubuntu focal/universe Translation-en [5124 kB]
    amazon-ebs: Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata [265 kB]
    amazon-ebs: Get:19 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [144 kB]
    amazon-ebs: Get:20 http://archive.ubuntu.com/ubuntu focal/multiverse Translation-en [104 kB]
    amazon-ebs: Get:21 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata [9136 B]
    amazon-ebs: Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [555 kB]
    amazon-ebs: Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [143 kB]
    amazon-ebs: Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [9924 B]
    amazon-ebs: Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [67.1 kB]
    amazon-ebs: Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [10.8 kB]
    amazon-ebs: Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [352 B]
    amazon-ebs: Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [243 kB]
    amazon-ebs: Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [93.4 kB]
    amazon-ebs: Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [6188 B]
    amazon-ebs: Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [15.1 kB]
    amazon-ebs: Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [3892 B]
    amazon-ebs: Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [480 B]
    amazon-ebs: Get:34 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [112 B]
    amazon-ebs: Get:35 http://archive.ubuntu.com/ubuntu focal-backports/restricted amd64 c-n-f Metadata [116 B]
    amazon-ebs: Get:36 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4012 B]
    amazon-ebs: Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [1448 B]
    amazon-ebs: Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [224 B]
    amazon-ebs: Get:39 http://archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B]
    amazon-ebs: Fetched 16.3 MB in 40s (409 kB/s)
    amazon-ebs: Reading package lists...
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
==> amazon-ebs: E: Unable to locate package ansible
==> amazon-ebs: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored after 2 minutes 2 seconds: Script exited with non-zero exit status: 100.Allowed exit codes are: [0]

==> Wait completed after 2 minutes 2 seconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 100.Allowed exit codes are: [0]

==> Builds finished but no artifacts were created.

Output from a successful build:

    $ ~/CICDServerSetup$ packer build ami.json
amazon-ebs: output will be in this color.

==> amazon-ebs: Prevalidating any provided VPC information
==> amazon-ebs: Prevalidating AMI Name: packer_build_test_2
    amazon-ebs: Found Image ID: ami-06e54d05255faf8f6
==> amazon-ebs: Creating temporary keypair: packer_5f7a8511-402e-5090-f078-da4b8716230c
==> amazon-ebs: Creating temporary security group for this instance: packer_5f7a8513-ab78-3566-d2cc-6528dbf60f85
==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-00074f05debcaa1d3
==> amazon-ebs: Waiting for instance (i-00074f05debcaa1d3) to become ready...
==> amazon-ebs: Using ssh communicator to connect: 35.167.183.26
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell578192723
    amazon-ebs: 'universe' distribution component is already enabled for all sources.
    amazon-ebs: Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
    amazon-ebs: Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
    amazon-ebs: Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
    amazon-ebs: Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [294 kB]
    amazon-ebs: Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB]
    amazon-ebs: Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
    amazon-ebs: Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [69.3 kB]
    amazon-ebs: Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [4696 B]
    amazon-ebs: Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [59.2 kB]
    amazon-ebs: Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [9856 B]
    amazon-ebs: Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [92.3 kB]
    amazon-ebs: Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [35.0 kB]
    amazon-ebs: Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [2956 B]
    amazon-ebs: Get:14 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [1252 B]
    amazon-ebs: Get:15 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [540 B]
    amazon-ebs: Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [116 B]
amazon-ebs: Get:17 http://archive.ubuntu.com/ubuntu focal/universe Translation-en [5124 kB]
amazon-ebs: Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata [265 kB]
amazon-ebs: Get:19 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [144 kB]
amazon-ebs: Get:20 http://archive.ubuntu.com/ubuntu focal/multiverse Translation-en [104 kB]
amazon-ebs: Get:21 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata [9136 B]
amazon-ebs: Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [555 kB]
amazon-ebs: Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [143 kB]
amazon-ebs: Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [9924 B]
amazon-ebs: Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [67.1 kB]
amazon-ebs: Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [10.8 kB]
amazon-ebs: Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [352 B]
amazon-ebs: Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [243 kB]
amazon-ebs: Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [93.4 kB]
amazon-ebs: Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [6188 B]
amazon-ebs: Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [15.1 kB]
amazon-ebs: Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [3892 B]
amazon-ebs: Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [480 B]
amazon-ebs: Get:34 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [112 B]
amazon-ebs: Get:35 http://archive.ubuntu.com/ubuntu focal-backports/restricted amd64 c-n-f Metadata [116 B]
amazon-ebs: Get:36 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4012 B]
amazon-ebs: Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [1448 B]
amazon-ebs: Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [224 B]
amazon-ebs: Get:39 http://archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B]
amazon-ebs: Fetched 16.3 MB in 7s (2179 kB/s)
amazon-ebs: Reading package lists...
amazon-ebs: Reading package lists...
amazon-ebs: Building dependency tree...
amazon-ebs: Reading state information...
amazon-ebs: The following additional packages will be installed:
amazon-ebs:   ieee-data python3-argcomplete python3-crypto python3-dnspython
amazon-ebs:   python3-jmespath python3-kerberos python3-libcloud python3-lockfile
amazon-ebs:   python3-netaddr python3-ntlm-auth python3-requests-kerberos
amazon-ebs:   python3-requests-ntlm python3-selinux python3-winrm python3-xmltodict
amazon-ebs: Suggested packages:
amazon-ebs:   cowsay sshpass python-lockfile-doc ipython3 python-netaddr-docs
amazon-ebs: The following NEW packages will be installed:
amazon-ebs:   ansible ieee-data python3-argcomplete python3-crypto python3-dnspython
amazon-ebs:   python3-jmespath python3-kerberos python3-libcloud python3-lockfile
amazon-ebs:   python3-netaddr python3-ntlm-auth python3-requests-kerberos
amazon-ebs:   python3-requests-ntlm python3-selinux python3-winrm python3-xmltodict
amazon-ebs: 0 upgraded, 16 newly installed, 0 to remove and 48 not upgraded.
amazon-ebs: Need to get 9643 kB of archives.
amazon-ebs: After this operation, 90.2 MB of additional disk space will be used.
amazon-ebs: Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-crypto amd64 2.6.1-13ubuntu2 [237 kB]

And then continues on to execute successfully and build the ami

Wheen

Answer 1

Try to add a 10 min sleep as the first provisioner. Ubuntu AMIs come with automatic updates on. So, whenever an instance is started, It will get updated itself.

Ramose

That's a might long delay for a short install process. Surely there must be another way? – Lascar

Using an arbitrary time is never a good solution. – Scheld

Answer 2

Hasicorp/Packer docs specifically addresses this and the fix is to add a provisioner that waits for the boot to complete.

{
  "type": "shell",
  "inline": [
    "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
  ]
}

Lascar

This is the correct answer and worked for me. Incredibly, with the latest version of packer in Dec 2022, this is still an issue. Packer should wait before full boot before allowing a provisioner to execute code on a linux machine. – Crying

Good answer. Wouldn't it be nice for Hashi to just build this in... – Marchesa

Hasicorp/Packer has updated their docs since this. The new snippet is: { "type": "shell", "inline": [ "cloud-init status --wait" ]} – Lascar

Answer 3

I've been plagued by this issue in Azure with Ubuntu 20.04 LTS I think it's probably the same underlying problem though. I tried all sorts of things, as soon as I added apt update as the very first provisioning step the build process become pretty solid. My daily build have worked for several weeks now. I run the commands below to get everything updated.

 apt update
 apt-get update
 apt-get -y install
 unattended-upgrades

Winonawinonah