AWS NVME mounted to different directory after reboot

Question

I'm trying to mount a few volumes to my instance and they are all NVME. I read that NVME volumes don't keeping their mapping the same, each time they are given serialized numbers randomly. The point is I need to keep the mapping consistent, it's for a db and 1 of the volumes suppose to keep the data. Now if I reboot the instance the volumes get mixed up, so it's possible that the volume that has the data will be mounted to a different directory and hence the db service starts and doesn't find any data.

Of course it happens also after creating an image, so I can't configure 1 instance and the spin up more using an image.

How can I force the mapping to be consistent? or stop using NVME? (I read this random serialization happens only with NVME)

Honor

Answer 1

You need to use device UUID. See my example below.

I have 3 disks, 8 GB, 10 GB and 12 GB.
They show as device nvme0n1 (8 GB), nvme1n1 (10 GB) and nvme2n1 (12 GB).

$ lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0  10G  0 disk
└─nvme1n1p1   259:6    0  10G  0 part /mnt/disk10
nvme2n1       259:1    0  12G  0 disk
└─nvme2n1p1   259:7    0  12G  0 part /mnt/disk12
nvme0n1       259:2    0   8G  0 disk
├─nvme0n1p1   259:3    0   8G  0 part /
└─nvme0n1p128 259:4    0   1M  0 part

Look that I have a file on disk of size 10 GB called /mnt/disk10/file10.txt.
Also a file on disk of size 12 GB called /mnt/disk12/file12.txt.

$ ls -l /mnt/*
/mnt/disk10:
total 0
-rw-r--r-- 1 root root 0 May  9 00:37 file10.txt

/mnt/disk12:
total 0
-rw-r--r-- 1 root root 0 May  9 00:38 file12.txt

My fstab file use UUID to refer those disks, as you can see below.

$ cat /etc/fstab
# Disk 8 GB
UUID=7b355c6b-f82b-4810-94b9-4f3af651f629     /           xfs    defaults,noatime  1   1
# Disk 10 GB
UUID=2b19004b-795f-4da3-b220-d531c7cde1dc     /mnt/disk10           xfs    defaults,noatime  0   0
# Disk 12 GB
UUID=1b18a2f2-f48f-4977-adf8-aa483e1fa91f     /mnt/disk12           xfs    defaults,noatime  0   0

If you want to know what is the UUID for each device, use blkid, as you can see below.

$ blkid
/dev/nvme1n1: PTUUID="2e6aaa33" PTTYPE="dos"
/dev/nvme1n1p1: UUID="2b19004b-795f-4da3-b220-d531c7cde1dc" TYPE="xfs" PARTUUID="2e6aaa33-01"
/dev/nvme2n1: PTUUID="10565c83" PTTYPE="dos"
/dev/nvme2n1p1: UUID="1b18a2f2-f48f-4977-adf8-aa483e1fa91f" TYPE="xfs" PARTUUID="10565c83-01"
/dev/nvme0n1: PTUUID="1760802e-28df-44e2-b0e0-d1964f72a39e" PTTYPE="gpt"
/dev/nvme0n1p1: LABEL="/" UUID="7b355c6b-f82b-4810-94b9-4f3af651f629" TYPE="xfs" PARTLABEL="Linux" PARTUUID="a5dcc974-1013-4ea3-9942-1ac147266613"
/dev/nvme0n1p128: PARTLABEL="BIOS Boot Partition" PARTUUID="dc255fff-03c6-40e6-a8dc-054ec864a155"

Now I will stop my machine, force change device order and start it again.
Look how the disk changes the device name, but they are still being mounted on the same path, with the same files on it.

Before: nvme0n1 (8 GB), nvme1n1 (10 GB) and nvme2n1 (12 GB).
Now: nvme0n1 (8 GB), nvme1n1 (12 GB) and nvme2n1 (10 GB).

$ lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0  12G  0 disk
└─nvme1n1p1   259:1    0  12G  0 part /mnt/disk12
nvme2n1       259:2    0  10G  0 disk
└─nvme2n1p1   259:3    0  10G  0 part /mnt/disk10
nvme0n1       259:4    0   8G  0 disk
├─nvme0n1p1   259:5    0   8G  0 part /
└─nvme0n1p128 259:6    0   1M  0 part

$ ls -l /mnt/*
/mnt/disk10:
total 0
-rw-r--r-- 1 root root 0 May  9 00:37 file10.txt

/mnt/disk12:
total 0
-rw-r--r-- 1 root root 0 May  9 00:38 file12.txt

UUID is an attribute from filesystem, so any time you create an filesystem it will generate an UUID. Also any time you generate an AMI or snapshot, the UUID is the same, as it belongs to filesystem, not to EBS volume.

Zygotene

Thanks for your answer @Azize. Yes that solve this issue. Would you know how to solve this for images. In this case, there is no way to know the devices' UUID since they will be created and attached when launching a new instance. My question: serverfault.com/questions/1062962/… – Threecolor

UUID is not tied to device, in fact it is related to filesystem when you format the device (partition). When you generate an AMI it will have all volumes exactly as it is when you generate the AMI, so every instance created from this AMI will have exactly the same UUID. No changes required. – Zygotene

It would be nice to know how to fetch this UUID during the first time boot and store it into fstab automatically. – Repugn

Boot volume is already configured with UUID on fstab when you create the instance. If you add more volumes when you create the instance, those extra volumes will not have a filesystem. You will need to do it, and configure fstab using UUID it generates. – Zygotene