How to create an .IMG image of a disc (sd card) without including free space?

Question

In Linux, we can do

dd if=/dev/sdb of=bckup.img

but if the disk is of 32GB with only 4GB used, the 32GB image file is waste of space-time. Is there any way or tool to create images with only valid data?

Bucephalus

Answer 1

The best thing to do is

1. Copy all the files from all the partitions preserving meta data

   mkdir -p myimage/partition1

   mkdir myimage/partition2

   sudo cp -rf --preserve=all /media/mount_point_partition1/* myimage/partition1/

   sudo cp -rf --preserve=all /media/mount_point_partition2/* myimage/partition2/

2. Extract the MBR

   sudo dd if=/dev/sdX of=myimage/mbr.img bs=446 count=1

   replace /dev/sdX with the corresponding device.

3. Partition the destination disk into partitions with sizes greater than copied data and should be of the same format and same flags using gparted. Google how to partition a disk.

4. Mount the freshly formatted and partitioned disk. On most computers, you just need to connect the disk and you can find the mounted partitions in /media folder.

5. Copy the previously copied data to destination partitions using following commands

   sudo cp -rf --preserve=all myimage/partition1/* /media/mount_point_partition1/

   sudo cp -rf --preserve=all myimage/partition2/* /media/mount_point_partition2/

6. Copy back the MBR

   sudo dd if=myimage/mbr.img of=/dev/sdX bs=446 count=1

Now njoy Ur new disk!

Bucephalus

Has anyone tried this and confirm that it works? Otherwise, I would supply only one of -r or -R to cp: they are the same switch. – Umeh

I've tried this and it worked. Made a clone of my raspberry pi's boot SD card. The card had 2 partitions, one fat and one ext4. I have dumped those with commands provided, then I dd'ed entire SD card with zeroes, created new partiion table, created my partitions which had different from previous parameters: size, spacimg, labels. And after that I restored it from dumps and it worked. – Consols

This works well, but you can also just resize partitions using gparted in one step. Backing up is still a good idea though. – Idomeneus

When you say MBR you probably mean Bootloader that is 446 bytes in size - amount you're extracting. The MBR actually is 512 bytes. – Folkrock

Answer 2

Pretty good and simple way to deal with this is simply pipe it via gzip, something like this:

# dd if=/dev/sdb | gzip > backup.img.gz

This way your image will be compressed and most likely unused space will be squeezed to almost nothing.

You would use this to restore such image back:

# cat backup.img.gz | gunzip | dd of=/dev/sdb

One note: if you had a lot of files which were recently deleted, image size may be still large (deleting file does not necessarily zeroes underlying sectors). You can wipe free space by creating and immediately deleting large file containing zeros:

# cd /media/flashdrive
# dd if=/dev/zero of=bigfile bs=1M     # let it run and quit by disk full error
# rm bigfile

Birdhouse

Is the free space in brand new sd card is zeros? – Bucephalus

most likely, but there is no 100% guarantee. – Birdhouse

I want to avoid this time consuming zero making process – Bucephalus

and ur solution still writes 32GB of data to sd card. a very time consuming process. – Bucephalus

Well, you can simply tar cvfz mybackup.tar.gz /media/flashdrive. It will capture everything except for bootable USB drives they will lose bootable status – Birdhouse

Its ok if it effects MBR. I can write MBR but I want all permissions, all hardlinks, all etc. Actually i installed ubuntu arm os on the sd card and u can guess that I want to avoid this installation process with my question. – Bucephalus

Yes, tar would lose MBR, and you will need to use boot-repair to make drive bootable again. But, other than that, tar should preserve everything and OS should be fully usable (but be sure to run tar as root!). – Birdhouse

zcat backup.img.gz | dd of=/dev/sdb, slightly neater than cat ... | gunzip | ... – Saltation

Does this skip the unallocated spaces in an SD card as well? For example, if only 4GB of 32GB large SD card is partitioned and used, will it skip the last 28GB of unused, unallocated space? – Illbehaved

@AdamLee: No. But if you use /dev/sdb1, it should. – Birdhouse

@Birdhouse Right, is there a way to get this working for multi-partition card? – Illbehaved

Yes, check my comment above using tar and boot-repair. – Birdhouse

You can usually zero out free space on modern sdcards using 'fstrim', this has the advantage of being many times faster, and also avoids flash wear.For fstrim support on VFAT partitions, you will need at least Linux kernel version 4.19. Unfortunately, many USB SD Card reader/writers don't support discard/trim. If you wish to try this, try fstrim -v /path-to-mounted-filesystem/. I use this during production image writing on an embedded air quality sensor which I work on professionally. – Palmate

Answer 3

Using the bs and count parameters of dd, you can limit the size of the image, as seen in step 2 of answer 1665017.

You may already know what size image you want to create. If not, you can get a good idea from df:

df -H --total /

Substitute / with a space-separated list of all the mount points relating to the disk partitions.

A more accurate way might be to use fdisk or your preferred partition editor and get busy with a calculator.

$ fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00057540

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        2048  186367  184320   90M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      186368 3667967 3481600  1.7G  5 Extended
/dev/mmcblk0p5      188416 3667967 3479552  1.7G 83 Linux

Total used space in bytes = end sector of last partition X sector size (here that's 3667967 x 512).

Total used space in GB = total used space in bytes / 1024³ (here that's 1.749023 GB).

If you decide, for example, that your image should be exactly 2 GB, the following command will do that:

dd if=/dev/mmcblk0 of=/path/to/pi_updated.img bs=1M count=2048

The resulting image will also include the random noise beyond the greatest extent of the last partition.

Don't forget to sudo the above commands if your account doesn't already have sufficient privileges.

For my purposes, I don't need an image that is perfectly trimmed down to the last bit of data so when the real size is 1.75 GB then a 2 GB image is near enough for me. This cuts out the other 6 GB (or 30 GB or whatever the device has spare) of unused space that would otherwise be in the image.

I have seen advice in many places that dd should not be performed on a mounted partition and I followed that because it seems intuitively correct; it does seem rather like trying to sketch yourself making a sketch in a mirror with the sketch you're making also visible in the sketch. It's a bit sketchy.

Hepplewhite

IMHO, while calculating, the sector size should be based one End+1 instead of End. In your case, it should be 3667968. – Unplaced

This will not work if partitions are not in order. Eg if empty space comes before the partition. – Sergu

Answer 4

After trying multiple different methods, I found the following article:

https://medium.com/platformer-blog/creating-a-custom-raspbian-os-image-for-production-3fcb43ff3630

It's created to shrink and resize (on first boot) a raspberry-pi image but could be easily adjusted for any other Linux distribution. I successfully got it working with Debian 9 on a custom arm based chip.

The rc.local created by the pishrink script first uses raspi-config to resize the rootfs and then falls back to a method using parted (which I had to install ahead of time on my machine). I commented out the section of code where raspi-config.

My sd-card image was shrunk from 15 GB to 1.1 GB. I flashed the shrunk sd-card image with etcher. Took less than 5 mins as opposed to over half an hour for the full 15 GB image

Moneyed

Answer 5

Sparse (Shrunken) Image Cloning Instructions:

Plug in bootable Ubuntu persistent live USB with image file in usbdata partition into Single-Board Computer such as UP² Board. Boot to USB (default persistent live option). Launch Terminal and type in the following command (single line):

sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress

Wait 15-20 minutes. When completed, shutdown and disconnect USB. Boot to internal eMMC drive

Creating the Image:

Create uncompressed sparse image file from internal eMMC and save to flash drive

sudo dd if=/dev/mmcblk0 bs=1M status=progress | cp --sparse=always /dev/stdin /media/ubuntu/usbdata/ubuntu.img

Extract uncompressed image file to internal eMMC

sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress

Create compressed image file from internal eMMC and save to Desktop

sudo dd if=/dev/mmcblk0 bs=1M status=progress | gzip > /home/user/Desktop/ubuntu.img.gz

Extract compressed image file to internal eMMC

sudo gzip -cd /home/user/Desktop/ubuntu.img.gz | dd of=/dev/mmcblk0 bs=1M status=progress

Sources: http://dustymabe.com/2012/11/15/create-a-disk-image-without-enough-free-space/ https://forum.up-community.org/discussion/1197/solved-cloning-the-image-of-an-up-board

To modify an existing Linux image or drive from within Linux, plug in drive or mount the image:

sudo mkdir /media/root
sudo mount -t ext4 -o loop,offset=537919488 /media/user/usbdata/ubuntu.img  /media/root

To determine the offset for your image or drive, run:

fdisk -l /media/user/usbdata/ubuntu.img

Then, multiply block-size of 512 bytes by the start-block of the Linux partition, 1050624. 512*1050624 = 537,919,488

Also mount other system folders:

for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /media/root$i; done

Then cd into root folder of img and chroot:

cd /media/root
sudo chroot .

Then you can update files, passwd, etc. To unmount, run:

for i in /dev /dev/pts /proc /sys /run; do sudo umount /media/root$i; done
sudo umount /media/root

Creating Ubuntu Persistent Live USB Image Follow these steps to flash a USB drive with a persistent live installation of Ubuntu: https://www.howtogeek.com/howto/14912/create-a-persistent-bootable-ubuntu-usb-flash-drive/

Belovo

Answer 6

If you have a big SD card 16 GB, 32 GB etc but you want to save space with backup you can use:

sudo apt-get install gnome-disk-utility

Open disk utility to check witch letter is your usb drive actually has:

gnome-disks

In my case a 32GB SD card with Raspbian image on it recognised as: /dev/sde

So I run with /dev/sde:

sudo dd bs=4M status=progress if=/dev/sde | gzip > \
/you-selected-full-path-here/raspberry-images/`date +%Y%m%d`_rpi_image_backup.gz

status=progress gives you progress bar indication
| gzip > compresses the 32 GB total size and not writing the empty space from the 32 GB
`date +%Y%m%d` writes today date in the filename

Output: 20190529_rpi_image_backup.gz

And the size is only 3.5GB. If you want to write this image to a new SD card use:

https://www.balena.io/etcher/

Also you can write this image made from 32 GB to a 16 GB or 8 GB disk, it is not complaining that the image is too large anymore.

Eventempered

I tried your steps for my 64 GB SD card which has 2 partitions. In total their size is about 7GB. However, after several minutes I see the process is not finished and the size of the created image is more than 25GB. I did again with gzio -9 and the result is almost the same. Why don't I have an 7GB image? – Anesthetist

Answer 7

My solution to make an image of a customized raspios:

DIR="./img"
DEVICE=/dev/sdb

# declare custom copy functions
copy() {
  echo "[$1] copy $2 to $3..."
  if [ "$1" == 'ext' ]; then
    sudo rsync -aHAXX \
      --no-inc-recursive \
      --numeric-ids \
      --filter='-x security.selinux' \
      --delete \
      --exclude={"/lost+found"} \
      --info=progress2 \
      $2 \
      $3
  else
    sudo rsync -rtD \
      --modify-window=1 \
      --no-inc-recursive \
      --delete \
      --exclude={"/lost+found"} \
      --info=progress2 \
      $2 \
      $3
  fi
}

# create working directories
mkdir -p $DIR/{mount,image}/{boot,rootfs}

# mount initial device partitions
sudo mount ${DEVICE}1 $DIR/mount/boot
sudo mount ${DEVICE}2 $DIR/mount/rootfs

# preserve partitions UUIDs
PTUUID=0x`sudo blkid -p $DEVICE | sed -e 's/.*PTUUID="\([0-9a-f]*\)".*/\1/'`
BOOTUUID=`sudo blkid -p ${DEVICE}1 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/' | tr -d '-'`
ROOTUUID=`sudo blkid -p ${DEVICE}2 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/'`

# copy contents from physical device to local folder
copy fat $DIR/mount/boot/   $DIR/image/boot/
copy ext $DIR/mount/rootfs/ $DIR/image/rootfs/
umount ${DEVICE}1
umount ${DEVICE}2

# show contents size
BOOTSIZE=`sudo du -hs $DIR/image/boot`
ROOTSIZE=`sudo du -hs $DIR/image/rootfs`
echo "boot is $BOOTSIZE"
echo "root is $ROOTSIZE"

# allocate the final .img file
# change size as needed
SIZE="5"
echo "allocating ${SIZE}G image..."
rm $DIR/$IMAGE
fallocate -l ${SIZE}G $DIR/$IMAGE

# create partitions
echo "formatting image..."
sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' << EOF | fdisk $DIR/$IMAGE > /dev/null
o          # create MBR partition scheme
n          # new partition
p          # primary
1          # partition #1
8192       # start block
532479     # end block
t          # define type
0c         # W95 FAT32 (LBA)
n          # new partition
p          # primary
2          # partition #2
532480     # start block
           # last end block
t          # define type
2          # for partition #2
83         # linux
x          # advanced menu
i          # change disk identifier
$PTUUID    # identifier from original fstab
r          # return to normal menu
w          # write changes
EOF

# locally mount the .img
echo "mounting image..."
LOOP=`sudo losetup -fP --show $DIR/$IMAGE`
echo "mounted image under $LOOP"
ls -la $LOOP*

echo "creating partition boot with FAT32"
sudo mkfs.fat \
  -F 32 \
  -S 512 \
  -i $BOOTUUID \
  -n boot \
  ${LOOP}p1

echo "creating partition rootfs with ext4"
sudo mkfs.ext4 \
  -b 4096 \
  -U $ROOTUUID \
  -L rootfs \
  ${LOOP}p2

echo "mounting image loops"
sudo mount ${LOOP}p1 $DIR/mount/boot
sudo mount ${LOOP}p2 $DIR/mount/rootfs

echo "copying contents to image"
copy fat $DIR/image/boot/   $DIR/mount/boot/
copy ext $DIR/image/rootfs/ $DIR/mount/rootfs/

echo "unmounting image"
unmount "$LOOP+" > /dev/null
echo "removing loop"
sudo losetup -d ${LOOP}

Liebig

imma going to try this :) but why the rsync not copying symbolic links, etc? – Afraid

it does: explainshell.com/… – Liebig

This script doesn't work without errors, at best, and is very dangerous at worst. It may overwrite data. At the very least, make sure you have enough free space, and note that this script needs around double the space. – Dedans