Implementing an update/upgrade system for embedded Linux devices

Question

I have an application that runs on an embedded Linux device and every now and then changes are made to the software and occasionally also to the root file system or even the installed kernel.

In the current update system the contents of the old application directory are simply deleted and the new files are copied over it. When changes to the root file system have been made the new files are delivered as part of the update and simply copied over the old ones.

Now, there are several problems with the current approach and I am looking for ways to improve the situation:

• The root file system of the target that is used to create file system images is not versioned (I don't think we even have the original rootfs).
• The rootfs files that go into the update are manually selected (instead of a diff)
• The update continually grows and that becomes a pita. There is now a split between update/upgrade where the upgrade contains larger rootfs changes.
• I have the impression that the consistency checks in an update are rather fragile if at all implemented.

Requirements are:

• The application update package should not be too large and it must also be able to change the root file system in the case modifications have been made.
• An upgrade can be much larger and only contains the stuff that goes into the root file system (like new libraries, kernel, etc.). An update can require an upgrade to have been installed.
  Could the upgrade contain the whole root file system and simply do a dd on the flash drive of the target?
• Creating the update/upgrade packages should be as automatic as possible.

I absolutely need some way to do versioning of the root file system. This has to be done in a way, that I can compute some sort of diff from it which can be used to update the rootfs of the target device.

I already looked into Subversion since we use that for our source code but that is inappropriate for the Linux root file system (file permissions, special files, etc.).

I've now created some shell scripts that can give me something similar to an svn diff but I would really like to know if there already exists a working and tested solution for this.

Using such diff's I guess an Upgrade would then simply become a package that contains incremental updates based on a known root file system state.

What are your thoughts and ideas on this? How would you implement such a system? I prefer a simple solution that can be implemented in not too much time.

Discontent

Good question +1 from my side – Paganism

Did u achieve this goal? Becuase i want to know something for your side , can u please help me in my question – Paganism

Answer 1

I believe you are looking wrong at the problem - any update which is non atomic (e.g. dd a file system image, replace files in a directory) is broken by design - if the power goes off in the middle of an update the system is a brick and for embedded system, power can go off in the middle of an upgrade.

I have written a white paper on how to correctly do upgrade/update on embedded Linux systems [1]. It was presented at OLS. You can find the paper here: https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf

[1] Ben-Yossef, Gilad. "Building Murphy-compatible embedded Linux systems." Linux Symposium. 2005.

Uticas

Thanks a lot for that paper, I will read it ASAP. I know, that the non atomic update is actually broken but it would still be better than the current situation that does no make any consitency checks and only extracts a tarbal. And I still need a way to version the rootfs. – Discontent

Please see my answer for this question: #5167726 It is based on the paper by gby, so I am including this as a comment. @Uticas It's great to see you on SO. I would greatly appreciate your comments on my referenced answer regarding the use of stamp files for detecting upgrade failures. – Embrace

the link to paper is broken, can you please update it? – Bugbee

Link is not broken anymore, though there is a certificate warning from the browser. – Dawkins

Could you fix the link to avoid a certificate warning? – Vociferate

@CraigMcQueen Sorry, no. It seems it is a mis-configuration on the server, which is not under my control. – Uticas

It would be great if your answer could cover the essential points of your paper, in case the link becomes broken or hard-to-get. – Vociferate

I get a 403 Forbidden You don't have permission to access /OLS/Reprints-2005/ben-yossef-Reprint.pdf on this server. Could someone with access to the link provide a summary here on SO since the link is not reliable? – Edrisedrock

Came across this PDF (kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf) by Ben Yossef and not sure if it's the same as the intended on in this post. – Edrisedrock

Answer 2

I absolutely agree that an update must be atomic - I have started recently a Open Source project with the goal to provide a safe and flexible way for software management, with both local and remote update. I know my answer comes very late, but it could maybe help you on next projects.

You can find sources for "swupdate" (the name of the project) at github.com/sbabic/swupdate.

Stefano

Jerald

Answer 3

Currently, there are quite a few Open Source embedded Linux update tools growing, with different focus each.

Another one that is worth being mentioned is RAUC, which focuses on handling safe and atomic installations of signed update bundles on your target while being really flexible in the way you adapt it to your application and environment. The sources are on GitHub: https://github.com/rauc/rauc

In general, a good overview and comparison of current update solutions you might find on the Yocto Project Wiki page about system updates:

https://wiki.yoctoproject.org/wiki/System_Update

Laconic

Answer 4

Atomicity is critical for embedded devices, one of the reasons highlighted is power loss; but there could be others like hardware/network issues.

Atomicity is perhaps a bit misunderstood; this is a definition I use in the context of updaters:

• An update is always either completed fully, or not at all
• No software component besides the updater ever sees a half installed update

Full image update with a dual A/B partition layout is the simplest and most proven way to achieve this.

For Embedded Linux there are several software components that you might want to update and different designs to choose from; there is a newer paper on this available here: https://mender.io/resources/Software%20Updates.pdf

File moved to: https://mender.io/resources/guides-and-whitepapers/_resources/Software%2520Updates.pdf

If you are working with the Yocto Project you might be interested in Mender.io - the open source project I am working on. It consists of a client and server and the goal is to make it much faster and easier to integrate an updater into an existing environment; without needing to redesign too much or spend time on custom/homegrown coding. It also will allow you to manage updates centrally with the server.

Morphology

Answer 5

You can journal an update and divide your update flash into two slots. Power failure always returns you to the currently executing slot. The last step is to modify the journal value. Non atomic and no way to make it brick. Even it if fails at the moment of writing the journal flags. There is no such thing as an atomic update. Ever. Never seen it in my life. Iphone, adroid, my network switch -- none of them are atomic. If you don't have enough room to do that kind of design, then fix the design.

Clouet

Your use of the term "atomic" seems confusing. – Vociferate