Why does EC2 instance not start correctly after resizing root EBS volume?

Question

I was using the instructions on https://matt.berther.io/2015/02/03/how-to-resize-aws-ec2-ebs-volumes/ and http://atodorov.org/blog/2014/02/07/aws-tip-shrinking-ebs-root-volume-size/ to move to a EBS volume with less disk space. In both cases, when I attached the shrinked EBS volume(as /dev/xdva or /dev/sda1 , neither works) to an EC2 instance and start it, it stops on its own with the message

State transition reason
Client.InstanceInitiatedShutdown: Instance initiated shutdown

Some more tinkering and I found that the new volume did not have BIOS boot partition. So I used gdisk to make one and copied the MBR from the original volume(that works and using which I can start instances) to the new volume. Now the instance does not terminate but I am not able to ssh into the newly launched instance.

What might be the reason behind this happening? How can I get more information(from logs/AWS Console etc) on why this is happening?

Subastral

what os are you using for your machine? do you have anything in the AWS EC2 Console -> Syslog ? – Huxham

Amazon Linux and the AWS EC2 console was empty. – Subastral

Answer 1

To shrink a GPT partioned boot EBS volume below the 8GB that standard images seem to use you can do the following: (a slight variation of the dd method from https://matt.berther.io/2015/02/03/how-to-resize-aws-ec2-ebs-volumes/ )

source disk is /dev/xvdf, target is /dev/xvdg

1. Shrink source partition

   $ sudo e2fsck -f /dev/xvdf1
   $ sudo resize2fs -M /dev/xvdf1
   

   Will print something like

   resize2fs 1.42.12 (29-Aug-2014)  
   Resizing the filesystem on /dev/xvdf1 to 257491 (4k) blocks.  
   The filesystem on /dev/xvdf1 is now 257491 (4k) blocks long.
   

   I converted this to MB, i.e. 257491 * 4 / 1024 ~= 1006 MB

2. copy above size + a bit more from device to device (!), not just partition to partition, because that includes both partition table & data in the boot partition

   $ sudo dd if=/dev/xvdf of=/dev/xvdg bs=1M count=1100
   

3. now use gdisk to fix the GPT partition on the new disk

   $ sudo gdisk /dev/xvdg
   

   You'll be greeted with roughly

   GPT fdisk (gdisk) version 0.8.10
   
   Warning! Disk size is smaller than the main header indicates! Loading
   secondary header from the last sector of the disk! You should use 'v' to
   verify disk integrity, and perhaps options on the experts' menu to repair
   the disk.
   Caution: invalid backup GPT header, but valid main header; regenerating
   backup header from main header.#
   
   Warning! One or more CRCs don't match. You should repair the disk!
   
   Partition table scan:
     MBR: protective
     BSD: not present
     APM: not present
     GPT: damaged
   
   ****************************************************************************
   Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
   verification and recovery are STRONGLY recommended.
   ****************************************************************************
   
   Command (? for help):
   

   The following is the keyboard input within gdisk. To fix the problems, the data partition that is present in the copied partition table needs to be resized to fit on the new disk. This means it needs to be recreated smaller and it's properties need to be set to match the old partition definition. Didn't test it so it's maybe not required to relocate the backup table to the actual end of the disk but I did it anyways:

   • go to extra expert options: x
   • relocate backup data structures to the end of the disk: e
   • back to main menu: m

   Now to fixing the partition size

   • print and note some properties of partition 1 (and other non-boot partitions if they exist):
     i
1
     Will show something like

     Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
     Partition unique GUID: DBA66894-D218-4D7E-A33E-A9EC9BF045DB
     First sector: 4096 (at 2.0 MiB)
     Last sector: 16777182 (at 8.0 GiB)
     Partition size: 16773087 sectors (8.0 GiB)
     Attribute flags: 0000000000000000
     Partition name: 'Linux'
     

   • now delete
     d
1
     and recreate the partition
     n
1
     Enter the required parameters. All defaults worked for me here (= press enter), when in doubt refer to partition information from above

     • First sector = 4096
     • Last sector = whatever is the actual end of the new disk - take the default here
     • type = 8300 (Linux)

   • The new partition's default name did not match the old one. So change it to the original One
     c
1
Linux (see Partition name from above)

   • Next thing to change is the partition's GUID
     x
c
1
DBA66894-D218-4D7E-A33E-A9EC9BF045DB (see Partition unique GUID, not the partition guid code above that)

   • That should be it. Back to main menu & print state
     m
i
1
     Will now print

     Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
     Partition unique GUID: DBA66894-D218-4D7E-A33E-A9EC9BF045DB
     First sector: 4096 (at 2.0 MiB)
     Last sector: 8388574 (at 4.0 GiB)
     Partition size: 8384479 sectors (4.0 GiB)
     Attribute flags: 0000000000000000
     Partition name: 'Linux'
     

     The only change should be the Partition size.

   • write to disk and exit
     w
y

4. grow filesystem to match entire (smaller) disk. The fist step shrunk it down to the minimal size it can fit

   $ sudo resize2fs -p /dev/xvdg1
   

5. We're done. Detach volume & snapshot it.

6. Optional step. Choosing proper Kernel ID for the AMI.

If you are dealing with PVM image and encounter following mount error in instance logs

Kernel panic - not syncing: VFS: Unable to mount root

when your instance doesn't pass startup checks, you may probably be required to perform this additional step.

The solution to this error would be to choose proper Kernel ID for your PVM image during image creation from your snapshot. The full list of Kernel IDs (AKIs) can be obtained here.

Do choose proper AKI for your image, they are restricted by regions and architectures!

Rebellious

Deletion/recreation doesn't affect actual data? Only partition table? – Enrica

@Enrica yes, the partition table doesn't know about the data in the filesystem, it just needs to know where the filesystems are in terms of byte boundaries, and you recreate that data – Rebellious

Added optional step to your solution which was absent in my sequence for successful resize. Thanks, dude! – Enrica

Thanks for that, I'm surprised it wasn't marked as an answer – Decrepitude

Exactly what I needed, thanks so much @Rebellious and Suncatcher! Wish there was a way for me to get you guys a coffee or something – Gley

Answer 2

The problem was with the BIOS boot partition. I was able to solve this by first initializing an instance with a smaller EBS volume. Then detaching the volume and attaching it to an instance whihc will be used to copyt the contents fromt he larger volume o the smaller volume. That created a BIOS boot partition which actually works. Simply creating a new one and copying the boot partition does not work.

Now following the steps outlined in any of the two links will help one shrink the volume of root EBS.

Subastral

Answer 3

Today, using UBUNTU doesn't work any other solution here. However, I found it:

1. For caution: snapshot the large volume (backup)
2. CREATE an instance IDENTICAL as possible such as LARGE volume works well. BUT with a SMALLER volume (desired size)
3. Detach his new volume and ATTACH the large volume (as /dev/sda1) and START instance
4. ATTACH the smaller new volume as /dev/sdf
5. LOG IN in new instance. Mount smaller volume on /mnt: sudo mount -t ext4 /dev/xvdf1 /mnt
6. DELETE everything on /mnt. Ignore WARNNG/error since /mnt can’t be deleted :) with sudo rm -rf /mnt
7. Copy entire / to smaller volume: sudo cp -ax / /mnt
8. Exit from instance and Stop it in AWS console
9. Detach BOTH volumes. Now, re-attach the smaller volume, IMPORTANT, as /dev/sda1
10. Start instance. LOG IN instance and confirm everything is ok with smaller volume
11. Delete large volume, delete large snapshot, create a new snapshot of smaller volume. END.

Ambrogino

Dude! Legend! This is the only set of steps that actually worked for my Ubuntu 20.04.1 EC2 instance. Other guides would render the instance un-bootable. – Traverse

You deserve a medal for this answer. This is by far the easiest way to shrink a root EBS volume and I can confirm that it works. – Stearin

Only this method worked with ubuntu-focal-20.04-amd64-server AMI, others produce unbootable volumes – Tedmund

Answer 4

The above procedures are not complete, missing steps:

1. Copy disk UUID
2. Install grub boot loader
3. Copy label

A more complete procedure can be found here:

https://medium.com/@m.yunan.helmy/decrease-the-size-of-ebs-volume-in-your-ec2-instance-ea326e951bce

This procedure is faster and more simple (no dd/resize2fs only rsync).

Tested with newer Nvme AWS disks.

Post any questions if you need help

Chalutz