How do the Linux kernel developers test their code locally and after they have it committed? Do they use some kind of unit testing and build automation? Test plans?
How is the Linux kernel tested?
Asked Answered
youtube.com/watch?v=L2SED6sewRw , somewhere, I cant remember exactly, but I think in the QA section this is being talked about. –
Coastland
Anders' link is great - a Google Tech Talk by one of the top kernel developers, Greg Kroah Hartman. He validates the answer given below by kernel developer @adobriyan. Greg notes the fun thing about the kernel - no good way to test without running it - hard to do unit tests etc - many permutations. "We rely on the development community to test. We want as many functional tests as we can get, and performance tests also." A link straight to the testing discussion is youtube.com/… –
Godfearing
With the popularity of VMs, wouldn't it be possible to automate this by building the kernel with a bunch of config permutations and trying to boot up on them? It wouldn't be a "unit" test by any means, but it could catch bugs. –
Nickname
@DanielKaplan: If you assume there's about 1000 motherboards that each has one of 10 CPUs, plus 3 of 1000 PCI devices, plus 3 of 1000 USB devices; and that the kernel has 1000 different possibly compile time options; then you're looking at about 1000*10*1000*999*9981000*999*998*1000 possible permutations to test. If you do a nice 8-hour burn in test for each permutation and have a pool of 100 servers to run 400 VMs in parallel at the same time; then by the time you've got 1 millionth of it tested the results will all be obsolete because someone changed the code and you have to start again. –
Encamp
There is a little discussion about unit tests on ycombinator. –
Nighttime
Well, having some unit tests in the kernel would still be good. The fact that most parts of the kernel are dependant on the hardware doesn't mean that the small testable part is not worth testing. The kernel reimplement big parts of the libc (as it is not usable in kernel space) which could be unit tested. Some functions in some modules could also be tested and I know some trivial bugs that could have been avoided if this was the case (active only on some hardware but in parts of code that were isolated enough to be testable). –
Donall
youtu.be/L2SED6sewRw?t=14m26s –
Vowell
The title of the referenced YouTube video is Greg Kroah Hartman on the Linux Kernel. 49 min 24 secs. 2008-06-11. Channel " Google TechTalks". Part of the abstract: "...describes the rate of development for the Linux kernel, and how the development model is set up to handle such a large and diverse developer population and huge rate of change. It will detail who is doing the work" –
Apocrypha
The Linux kernel has a heavy emphasis on community testing.
Typically, any developer will test their own code before submitting, and quite often they will be using a development version of the kernel from Linus, or one of the other unstable/development trees for a project relevant to their work. This means they are often testing both their changes and other people's changes.
There tends not to be much in the way of formal test plans, but extra testing may be asked for before features are merged into upstream trees.
As Dean pointed out, there's also some automated testing: The Linux Test Project and the kernel Autotest (good overview).
Developers will often also write automated tests targeted to test their change, but I'm not sure there's a (often used) mechanism to centrally collect these ad hoc tests.
It depends a lot on which area of the kernel is being changed of course - the testing you'd do for a new network driver is quite different to the testing you'd do when replacing the core scheduling algorithm.
+1, half the battle is simply not breaking something that drivers depend on, hence the persistence of the BKL over the years. The other thing to consider is testing many sub systems on many different architectures, which is only practically feasible with the kind of community abuse, err testing, that Linux receives. –
Decontrol
Naturally, the kernel itself and its parts are tested prior to the release, but these tests cover only the basic functionality. There are some testing systems which perform testing of Linux Kernel:
The Linux Test Project (LTP) delivers test suites to the open source community that validate the reliability and stability of Linux. The LTP test suite contains a collection of tools for testing the Linux kernel and related features.
Autotest—a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing, and other general user space program testing under Linux platforms. It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.
Also there are certification systems developed by some major GNU/Linux distribution companies. These systems usually check complete GNU/Linux distributions for compatibility with hardware. There are certification systems developed by Novell, Red Hat, Oracle, Canonical, and Google.
There are also systems for dynamic analysis of the Linux kernel:
Kmemleak is a memory leak detector included in the Linux kernel. It provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak.
Kmemcheck traps every read and write to memory that was allocated dynamically (i.e., with kmalloc()). If a memory address is read that has not previously been written to, a message is printed to the kernel log. It is also is a part of the Linux kernel.
Fault Injection Framework (included in the Linux kernel) allows for infusing errors and exceptions into an application's logic to achieve a higher coverage and fault tolerance of the system.
How do the Linux kernel developers test their code locally and after they have it committed?
Do they use some kind of unit testing and build automation?
In the classic sense of words, no.
For example, Ingo Molnar is running the following workload:
- build a new kernel with a random set of configuration options
- boot into it
- go to 1
Every build fail, boot fail, bug or runtime warning is dealt with. 24/7. Multiply by several boxes, and one can uncover quite a lot of problems.
Test plans?
No.
There may be a misunderstanding that there is a central testing facility, but there is none. Everyone does what he/she wants.
Given the existence of sites such as this and this I also would question the validity of this answer. –
Flyblown
Source to what statement exactly? If it's about Ingo's setup, it's common knowledge on linux-kernel. –
Wellrounded
I think the core of adobriyan's answer "there is central testing facility, there is none." is about right. However different groups do varying levels of testing, it's not as though the kernel is completely untested. –
Drinker
I think both SUSE and RedHat in addition to testing their own kernels, test vanilla often. There is no central testing per se, but there is a testing going on nevertheless - by the major users of Linux. Otherwise the comment stands. Were it written less sarcastically I would have even modded it up. –
Tailback
Errr, do you all people realize that Alexey Dobriyan is a Linux kernel developer? –
Sox
@Sox Indeed! Search for Alexey Dobriyan on linux kernel mailing list: 4,371 matching articles –
Godfearing
As another kernel developer, I must say this is the most honest answer to the question: the kernel is NOT tested in the classic sense, simply because it's impossible. There are more combinations of configuration and hardware than available developer time to test. Very few people has the required skills to test certain devices, and in some cases very few people actually own certain devices. –
Wealthy
@EzequielGarcia What makes it impossible? –
Dupuis
@inf3rno: Impossible to test fully across every possible combination of config options, hardware, and compiler version. The number of combinations is enormous. The ad-hoc testing by developers on their own machines is somewhat reasonable for the core parts of the kernel, though. –
Emilia
@PeterCordes Sure, but unit testing is possible and if there is a standard(ish) API for the component it is possible to test against the API with integration tests written by the API developers. –
Dupuis
In-tree tools
A good way to find test tools in the kernel is to:
make helpand read all targets- look under tools/testing
In v4.0, this leads me to:
kselftest under tools/testing/selftests. Run with
make kselftest. Must be running built kernel already. See also: Documentation/kselftest.txt , https://kselftest.wiki.kernel.org/ktest under tools/testing/ktest. See also: http://elinux.org/Ktest , http://www.slideshare.net/satorutakeuchi18/kernel-auto-testbyktest
Static analysers section of
make help, which contains targets like:checkstack: Perl: what does checkstack.pl in linux source do?coccicheckfor Coccinelle (mentioned by askb)
Kernel CI
https://kernelci.org/ is a project that aims to make kernel testing more automated and visible.
It appears to do only build and boot tests (TODO how to test automatically that boot worked Source should be at https://github.com/kernelci/).
Linaro seems to be the main maintainer of the project, with contributions from many big companies: https://kernelci.org/sponsors/
Linaro Lava
http://www.linaro.org/initiatives/lava/ looks like a CI system with focus on development board bringup and the Linux kernel.
ARM LISA
https://github.com/ARM-software/lisa
Not sure what it does in detail, but it is by ARM and Apache Licensed, so likely worth a look.
Demo: https://www.youtube.com/watch?v=yXZzzUEngiU
Step debuggers
Not really unit testing, but may help once your tests start failing:
- QEMU + GDB: https://mcmap.net/q/15277/-how-is-linux-kernel-live-debugging-done-and-what-tools-are-used
- KGDB: https://mcmap.net/q/15277/-how-is-linux-kernel-live-debugging-done-and-what-tools-are-used
My own QEMU + Buildroot + Python setup
I also started a setup focused on ease of development, but I ended up adding some simple testing capabilities to it as well: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/8217e5508782827320209644dcbaf9a6b3141724#test-this-repo
I haven't analyzed all the other setups in great detail, and they likely do much more than mine, however I believe that my setup is very easy to get started with quickly because it has a lot of documentation and automation.
could you please also take a look at and potentially contribute to this –
Avicenna
@JhonnyS thanks for the ping. I'm afraid I'm not that familiar with uboot, so I don't have any suggestions besides the reading source/pinging respective upstreams :-) This might be related: titanwolf.org/Network/Articles/… –
Seta
thanks for the reference. Let me try that. –
Avicenna
It’s not very easy to automate kernel testing. Most Linux developers do the testing on their own, much like adobriyan mentioned.
However, there are a few things that help with debugging the Linux Kernel:
- kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
- dmesg: Definitely the place to look for information about what happened during the kernel boot and whether it works/doesn't work.
- Kernel Instrumentation: In addition to printk's (and an option called 'CONFIG_PRINTK_TIME' which allows you to see (to microsecond accuracy) when the kernel output what), the kernel configuration allows you to turn on a lot of tracers that enable them to debug what is happening.
Then, developers usually have others review their patches. Once the patches are reviewed locally and seen not to interfere with anything else, and the patches are tested to work with the latest kernel from Linus without breaking anything, the patches are pushed upstream.
Here's a nice video detailing the process a patch goes through before it is integrated into the kernel.
In addition to the other answers, this emphasise more on the functionality testing, hardware certification testing and performance testing the Linux kernel.
A lot of testing actually happen through scripts, static code analysis tools, code reviews, etc. which is very efficient in catching bugs, which would otherwise break something in the application.
Sparse – An open-source tool designed to find faults in the Linux kernel.
Coccinelle is another program does matching and transformation engine which provides the language SmPL (Semantic Patch Language) for specifying desired matches and transformations in C code.
checkpatch.pl and other scripts - coding style issues can be found in the file Documentation/CodingStyle in the kernel source tree. The important thing to remember when reading it is not that this style is somehow better than any other style, just that it is consistent. This helps developers easily find and fix coding style issues. The script scripts/checkpatch.pl in the kernel source tree has been developed for it. This script can point out problems easily, and should always be run by a developer on their changes, instead of having a reviewer waste their time by pointing out problems later on.
I would imagine they use virtualization to do quick tests. It could be something like QEMU, VirtualBox or Xen, and some scripts to perform configurations and automated tests.
Automated testing is probably done by trying either many random configurations or a few specific ones (if they are working with a specific issue). Linux has a lot of low-level tools (such as dmesg) to monitor and log debug data from the kernel, so I imagine that is used as well.
You are definitely right. When I did my kernel module development, I heavily depended on VirtualBox + KGDB to LINE-BY-LINE trace the kernel execution. Yes, gdb to see the whole kernel execute line-by-line is really cool. Same with Valerie Aurora, famous kernel developer, eg: youtube.com/watch?v=Tach2CheAc8. Inside the video u can see how she use UserModeLinux virtualization to step through the kernel. –
Curley
As far as I know, there is an automatically performance regression check tool (named lkp/0 day) running/funding by the Intel. It will test each valid patch sent to the mailing list and check the scores changed from different microbenchmarks such as hackbench, fio, unixbench, netperf, etc.
Once there is a performance regression/improvement, a corresponding report will be sent directly to the patch author and a Cc related maintainers.
adobriyan mentioned Ingo's loop of random configuration build testing. That is pretty much now covered by the 0-day test bot (aka kbuild test bot). A nice article about the infrastructure is presented here: Kernel Build/boot testing
The idea behind this set-up is to notify the developers ASAP so that they can rectify the errors soon enough (before the patches make it into Linus' tree in some cases as the kbuild infrastructure also tests against maintainer's subsystem trees).
Once after contributors submit their patch files and after making a merge request, Linux gatekeepers are checking the patch by integrating and reviewing it. Once it succeeds, they will merge the patch into the relevant branch and a make new version release.
The Linux Test Project is the main source which provides test scenarios (test cases) to run against the kernel after applying patches. This may take around 2 ~ 4 hours, and it depends.
Please note regarding the file system of the selected kernel is going to test against. Example: ext4 generates different results against ext3 and so on.
Kernel Testing procedure.
- Get latest kernel source from the repository (The Linux Kernel Archives or GitHub)
- Apply the patch file (using a diff tool)
- Build the new kernel.
- Test against test procedures in LTP (Linux Test Project)
Things have changed quite a bit since the most upvoted answers were written, now there is the Continuous Kernel Integration Project which has added a great deal of automation to the process. The scale of their efforts are really quite impressive. There is a good presentation on the project here, "Abusing Gitlab CI to Test Kernel Patches", which provides a great deal of information about the process and the efforts to improve it.
I had done Linux kernel compilation and done some modifications for Android (Android 6.0 (Marshmallow) and Android 7.0 (Nougat)) in which I use Linux version 3. I cross-compiled it on a Linux system, debugged the errors manually and then ran its boot image file in Android and checked if it was going in a loop-hole or not. If it runs perfect then it means it is compiled perfectly according to system requirements.
Note: The Linux kernel will change according to requirements which depend on system hardware
© 2022 - 2024 — McMap. All rights reserved.