Linux BTF: bpftool: Failed to get EHDR from /sys/kernel/btf/vmlinux

Question

I am trying to start with BPF CO:RE Development. Using Ubuntu 20.04 LTS in a VM, I needed to recompile the kernel and install pahole (from apt install dwarves) so that BTF is enabled (I set CONFIG_DEBUG_FS=y and CONFIG_DEBUG_INFO_BTF=y).

So my setup is:

• Ubuntu 20.04
• Kernel 5.4.0-90-generic
• bpftool --version: /usr/lib/linux-tools/5.4.0-90-generic/bpftool v5.4.148

/sys/kernel/btf/vmlinux exists and can be read out with cat.

But bpftool shows the following error:

$ sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c

libbpf: failed to get EHDR from /sys/kernel/btf/vmlinux
Error: failed to load BTF from /sys/kernel/btf/vmlinux: Unknown error -4001

From https://github.com/libbpf/libbpf/blob/master/src/libbpf.h it looks like it is LIBBPF_ERRNO__FORMAT, /* BPF object format invalid */ but I can not find out what's wrong.

Does anybody know where the mistake might be?

Thanks in advance!

EDIT: Added bpftool version

Christi

What's your version of bpftool? (bpftool version). Have you tried with the latest version? – Fable

It is v5.4.148, Where do I see which is the latest version? – Christi

Latest version is shipped with latest kernel sources :) If you have a local clone of the kernel sources you can build bpftool from there. But note I don't know if using a newer version will fix the issue, I'm just suggesting it might be worth checking. – Fable

As I am not using the latest kernel, building bpftool from the kernel sources I used did not change anything. I can download the latest kernel and give it a try, but still it should be working with the bpftool shipped with the kernel somehow I guess :/ – Christi

Did you try with the raw format? – Hageman

Yes, raw does not change anything. – Christi

Answer 1

You need to update bpftool to support a fallback to reading BTF as raw data if the input file is not an object file. The minimum bpftool version required is v5.5 as that's the Linux release where the patch landed. In general, I would recommend to always use the latest bpftool version as there are no backports.

Hageman

Hence the initial advice. Thanks pchaigno for finding the related commit! +1 – Fable

Thank you! :D Another question: What is the official way to get the vmlinux.h before bpftool 5.5? – Christi

I guess it would be using libbpf in your application, just like bpftool does after the patch. I'm not aware of any other command-line tools for dumping BTF information at the moment. – Fable

Thanks! How about pahole, which can also generate a (slightly different! :/) vmlinux.h? – Christi

Ah maybe pahole. I don't remember. Can it dump BTF, or just produce it as binary data? – Fable

Answer 2

Update: It looks like bpftool only accepts a ELF-file with the compiled runnning kernel in it, but my /sys/kernel/btf/vmlinux is not:

$ file /sys/kernel/btf/vmlinux 
/sys/kernel/btf/vmlinux: data

Same for /boot/vmlinuz:

$ sudo file /boot/vmlinuz-5.4.0-90-generic 
/boot/vmlinuz-5.4.0-90-generic: Linux kernel x86 boot executable bzImage, version 5.4.0-90-generic (root@elde-dev) #101+test1 SMP Tue Nov 23 16:38:41 UTC 2021, RO-rootFS, swap_dev 0xD, Normal VGA

Does anybody know why my /sys/kernel/btf/vmlinux does not show the right format?

I found this workaround:
Using this script (https://elixir.bootlin.com/linux/latest/source/scripts/extract-vmlinux) as suggested here (https://unix.stackexchange.com/questions/610672/where-is-the-linux-kernel-elf-file-located) I could get the "working" vmlinux-file which then could be read by bpftool. But this can not really be the right way for BPF CO:RE I guess... Also, in all the tutorials, bpftool is used directly with /sys/kernel/btf/vmlinux.
So why do I get the wrong format?

EDIT: As suggested above, just downoad the newest linux kernel, compile bpftool from there and use that.

Christi

Good that you found a workaround. For what it's worth, file /sys/kernel/btf/vmlinux also returns data on my setup where bpftool can read it successfully. I'm not sure what went wrong in your case though, sorry. – Fable