Linux-Based Firmware, how to implement a good way to update?

Question

I'm developing a linux-based appliance using an alix 2d13.

I've developed a script that takes care of creating an image file, creating the partitions, installing the boot loader (syslinux), the kernel and the initrd and, that takes care to put root filesystem files into the right partition.

Configuration files are on tmpfs filesystem and gets created on system startup by a software that reads an XML file that resides on an own partition.

I'm looking a way to update the filesystem and i've considered two solutions:

• the firmware update is a compressed file that could contain kernel, initrd and/or the rootfs partition, in this way, on reboot, initrd will takes care to dd the rootfs image to the right partition;
• the firmware update is a compressed file that could contain two tar archives, one for the boot and one for the root filesystem.

Every solution has its own advantages: - a filesystem image will let me to delete any unused files but needs a lot of time and it will kill the compact flash memory fastly; - an archive is smaller, needs less time for update, but i'll have the caos on the root filesystem in short time.

An alternative solution could be to put a file list and to put a pre/post update script into the tar archive, so any file that doesn't reside into the file list will be deleted.

What do you think?

Tude

Answer 1

I used the following approach. It was somewhat based on the paper "Building Murphy-compatible embedded Linux systems," available here. I used the versions.conf stuff described in that paper, not the cfgsh stuff.

• Use a boot kernel whose job is to loop-back mount the "main" root file system. If you need a newer kernel, then kexec into that newer kernel right after you loop-back mount it. I chose to put the boot kernel's complete init in initramfs, along with busybox and kexec (both statically linked), and my init was a simple shell script that I wrote.
• One or more "main OS" root file systems exist on an "OS image" file system as disk image files. The boot kernel chooses one of these based on a versions.conf file. I only maintain two main OS image files, the current and fall-back file. If the current one fails (more on failure detection later), then the boot kernel boots the fall-back. If both fail or there is no fall-back, the boot kernel provides a shell.
• System config is on a separate partition. This normally isn't upgraded, but there's no reason it couldn't be.
• There are four total partitions: boot, OS image, config, and data. The data partition is for user application stuff that is intended for frequent writing. boot is never mounted read/write. OS image is only (re-)mounted read/write during upgrades. config is only mounted read/write when config stuff needs to change (hopefully never). data is always mounted read/write.
• The disk image files each contain a full Linux system, including a kernel, init scripts, user programs (e.g. busybox, product applications), and a default config that is copied to the config partition on the first boot. The files are whatever size is necessary to fit everything in them. As long I allowed enough room for growth so that the OS image partition is always big enough to fit three main OS image files (during an upgrade, I don't delete the old fall-back until the new one is extracted), I can allow for the main OS image to grow as needed. These image files are always (loop-back) mounted read-only. Using these files also takes out the pain of dealing with failures of upgrading individual files within a rootfs.
• Upgrades are done by transferring a self-extracting tarball to a tmpfs. The beginning of this script remounts the OS image read/write, then extracts the new main OS image to the OS image file system, and then updates the versions.conf file (using the rename method described in the "murphy" paper). After this is done, I touch a stamp file indicating an upgrade has happened, then reboot.
• The boot kernel looks for this stamp file. If it finds it, it moves it to another stamp file, then boots the new main OS image file. The main OS image file is expected to remove the stamp file when it starts successfully. If it doesn't, the watchdog will trigger a reboot, and then the boot kernel will see this and detect a failure.
• You will note there are a few possible points of failure during an upgrade: syncing the versions.conf during the upgrade, and touching/removing the stamp files (three instances). I couldn't find a way to reduce these further and achieve everything I wanted. If anyone has a better suggestion, I'd love to hear it. File system errors or power failures while writing the OS image could also occur, but I'm hoping the ext3 file system will provide some chance of surviving in that case.

Threequarter

i like your solution a lot, in the end i did something very similar but without boot failure detection but i think that i'll implement something similar: - i will reserve a sector for the boot failure detection writing in it using direct io and sync operations - the boot kernel will start an initrd that will report a try-to-boot, with timestamp and mount the os image - during the boot process, the new initrd will report, step by step with timestamps, the boot phases to identify problems - in the end will report in the boot failure detection sector "all done" – Tude

For the upgrade I prefer to download the tarball in the data partition and, at the reboot, if the first initrd detects the upgrade tarball, it will handle the tarball (may be useful add two special files to identify files to delete and to execute an upgrade script). After the upgrade the first initrd will kexec to the new kernel, reporting the boot as upgrade (so the tarball can be deleted). The tarball, for security reasons, need to be signed with a private key. To report errors it could use the previous disk sector (boot failure detection), it just need to do sync write to avoid problems – Tude

I like your "status sector" idea. At least it moves some of the failure points to a single disk location, and if that sector fails, you could conceivably have spare sectors and you could attempt a recovery, or at least offer a "recovery mode." – Threequarter

One reason I did the upgrade from within the main OS is that I am allowing for the upgrade script to be anything (it could just update a config file -- it doesn't have to do the main OS). I wanted to keep the boot kernel simple, so it never has to change once released (though even some limited upgrade is possible there -- since kexec starts over from scratch, it runs the (possibly) newer boot kernel). Putting as much of the upgrade smarts within the upgrade script should help me to achieve that. – Threequarter

But in this way, don't you risk to have problems changing used files? While the stuff that is loaded in memory it is ok all the other stuff it isn't: in any instant an executable can dinamically load a shared library and everything could go crazy. May be that supporting both methods is the best thing: - simply updates, just config files, system rules or stuff like this; - complex updates, executable on the disk and similar. You can avoid that kind of problems just stopping the service/exacutable and all the related apps but this may became too complex – Tude

The update process does not impact any running processes (not yet anyway, since I'm not updating config files). And I can't modify any files in the main OS, since it's loop-back mounted read-only. All that is done during the upgrade is the new main OS image is extracted (it's just another file on the OS image file system), the versions.conf file is updated, and the stamp file is written. Then the reboot happens, and the new main OS image is selected by the boot kernel. – Threequarter

Sorry, I missed that you were using a file loop-back mounted as rootfs, but while this ensure more stability it requires a lot more space (at least the double, more if you have an history of os images). – Tude

Shameless plug - I hope this stack exchange takes off. We could use more committers: area51.stackexchange.com/proposals/50195/… – Threequarter

Answer 2

You can have a seperate partition for update(Say Side1/Side2). The existing kernel,rootfs is in Side1, then put the update in Side2 and switch. By this you can reduce wear leveling and increase the life but the device gets costlier.

Piston

Answer 3

You can quick format the partitions before extracting the tar files. Or go with the image solution but use the smallest possible image and after dd do a filesystem resize (although that is not necessary for readonly storage)

Melodic

mmm, if i do a dd i really don't need to do a resize because the filesystem would know it's size ... what i really i would need is to resize the partition in mbr. Anyway i can avoid resize doing bigger partition (actually i need less than 30mb so a 64mb partition would be fine). I'll go for the tar+pre/post update scripts+file list solution probably – Tude