Why is this kernel module marked at permanent on 2.6.39

Question

When I load this module:

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("Dual BSD/GPL");

static int hello_init(void) {
  printk("<1> Hello world!\n");
  return 0;
}

static void hello_exit(void) {
  printk("<1> Bye, cruel world\n");
}


module_init(hello_init);
module_exit(hello_exit);

(From http://www.freesoftwaremagazine.com/articles/drivers_linux?page=0,2 )

The module get marked as [permanent] in lsmod and can't be unloaded, on 2.6.39-02063904-generic (from the Ubuntu PPA). But it works fine on the default 2.6.38 kernel. (Both on Ubuntu 11.04 x86).

What has changed in 2.6.39? and what do I need to change in my code?

I was trying to isolate a more complicated problem when I ran into this issue.

EDIT:

Following a suggestion from an answer I edited the code to add __init and __exit (hello3.c):

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("Dual BSD/GPL");

static int __init hello_init(void) {
  printk("<1> Hello world!\n");
  return 0;
}

static void __exit hello_exit(void) {
  printk("<1> Bye, cruel world\n");
}

module_init(hello_init);
module_exit(hello_exit);

The build output:

make -C /lib/modules/2.6.39-02063904-generic/build M=/home/douglas/kernelmod modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.39-02063904-generic'
Building with KERNELRELEASE = 2.6.39-02063904-generic
  CC [M]  /home/douglas/kernelmod/hello3.o
  Building modules, stage 2.
Building with KERNELRELEASE = 2.6.39-02063904-generic
  MODPOST 8 modules
  CC      /home/douglas/kernelmod/hello3.mod.o
  LD [M]  /home/douglas/kernelmod/hello3.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.39-02063904-generic'

EDIT2:

hello3.mod.c:

#include <linux/module.h>
#include <linux/vermagic.h>
#include <linux/compiler.h>

MODULE_INFO(vermagic, VERMAGIC_STRING);

struct module __this_module
__attribute__((section(".gnu.linkonce.this_module"))) = {
 .name = KBUILD_MODNAME,
 .init = init_module,
#ifdef CONFIG_MODULE_UNLOAD
 .exit = cleanup_module,
#endif
 .arch = MODULE_ARCH_INIT,
};

static const struct modversion_info ____versions[]
__used
__attribute__((section("__versions"))) = {
    { 0xbe4b3e92, "module_layout" },
    { 0xb4390f9a, "mcount" },
    { 0x5e3b3ab4, "printk" },
};

static const char __module_depends[]
__used
__attribute__((section(".modinfo"))) =
"depends=";


MODULE_INFO(srcversion, "D2A869459874C22AB265981");

Also

# grep CONFIG_MODULE_UNLOAD /boot/config-2.6.39-02063904-generic 
CONFIG_MODULE_UNLOAD=y

EDIT3:

More interestingly it doesn't happen with a vanilla kernel I've compiled myself - that loads and unloads modules fine.

EDIT4:

I installed the Oneiric beta 2 build on a VM, and that 3.0.0-11 kernel doesn't have any problem either. So it appears to be limited to the Ubuntu Vanilla PPA kernels. That won't be much fun to resolve.

Ararat

Could you also post the contents of hello3.mod.c? – Rothmuller

@Rothmuller I've added hello3.mod.c and checked the config has CONFIG_MODULE_UNLOAD set (at least in the /boot copy - /proc/config.gz isn't enabled). – Ararat

Also module unload must work, since I can load/unload cifs. – Ararat

Ok, given all this, I haven't the faintest idea why this would fail. You may still want to verify that /lib/modules/2.6.39-02063904-generic/build/.config (ie. the configuration you're building against) matches the one in /boot and /proc/config.gz (if existing) – Rothmuller

@Rothmuller .config file also has CONFIG_MODULE_UNLOAD set. – Ararat

@Rothmuller - doesn't happen on a vanilla 2.6.39 kernel I compiled on that machine. I'll investigate the Ubuntu kernel line I guess. – Ararat

Answer 1

So, after consultation with Canonical, I know what the problem is:

Ubuntu mainline builds are built with the Hardy tool chain, and the 11.04 and 11.10 tool chains are incompatible for building out-of-tree kernel modules.

Ararat

Answer 2

"struct module" layout depends on HAVE_JUMP_LABEL define which depends on CC_HAVE_ASM_GOTO define which depends on the gcc-goto.sh script result which depends on gcc version being used. When there is a mismatch, the module exit callback (destructor) gets value of NULL which results in module being marked as [permanent].

Iamb

Wow. Did you just know that, or did you research it? In the end I just stopped using PPA kernels for my kernel module development/testing... – Ararat

I've been hunting the issue because it was affecting a code that I'm taking care of. First I tried to build the module using kbuild and it worked. Then I shifted more and more stuff to the external build system, until I found that it is the compilation of the modpost generated C file. Then I took C compiler options one by one. – Iamb

Answer 3

As far as I can tell from the kernel sources, a module is marked permanent if it has an init function, but lacks an exit function.

I'm not entirely sure in this case, but you may want to mark your init and exit functions with __init and __exit respectively. (also, pay attention to any warnings emitted by modpost)

Rothmuller

__init and __exit designate section for the linker to put the function code in so I don't believe it has anything to do with that – Shawanda

@gby: True, that shouldn't matter. as far as I can tell, assuming the module is properly compiled, the module_exit() macro should create an alias to cleanup_module(). It's possible that modpost isn't seeing it from some reason though, hence the suggestion to check the modpost warnings – Rothmuller

@Rothmuller I've tried adding __init and __exit and added the build output - No errors that I can see. – Ararat

@Rothmuller Turns out the Ubuntu mainline builds are built with Hardy tool chain, which is incompatible with 11.04 and 11.10 built out-of-tree kernel modules. – Ararat

Answer 4

This issue is due to elder gcc compiler generate wrong module binary format and cause the module exit function can not get proper unload inforamtion from kernel.

You may check if your gcc versioni is 4.4, if that, changing to use 4.6 and problem will fixed:

gcc --version

if 4.4 version, remove the symbol link of /usr/bin/gcc and relink it to /usr/bin/gcc-4.6. The module remove should works after you recompile it.

Canonicate

This is quite an old question: the issue turned out to be that the kernels were build with the hardy 8.04 tool chain, vs. the 11.04 toolchain, so actually the other way round - the newer gcc didn't work, and the older one would have. – Ararat

8.04 gcc = 4.2.3, 11.04 gcc = 4.5.2 – Ararat

oh, thanks, I didn't notice the kernel is quiet old in this question. My kernel is 3.15.* and has the similar [permanent] problem when using the 4.4 gcc compiler. – Canonicate