Is there any method to run perf under WSL?

Question

When I wanted to run perf under WSL, I met the follow question:

WARNING: perf not found for kernel 4.4.0-18362

You may need to install the following packages for this specific kernel:

linux-tools-4.4.0-18362-Microsoft

linux-cloud-tools-4.4.0-18362-Microsoft

You may also want to install one of the following packages to keep up to date:

linux-tools-Microsoft

linux-cloud-tools-Microsoft

But I can't find packages called linux-tools-4.4.0-18362-Microsoft or linux-cloud-tools-4.4.0-18362-Microsoft. I guess the package names are generated automatically.

I also tried to use perf in docker container. However, docker container use the same kernel as the hosts.

Is there any method to run perf under WSL?

---

I heard that perf can be used in WSL2. But after I upgraded to WSL2, it shows the similar error message:

WARNING: perf not found for kernel 4.19.84-microsoft

You may need to install the following packages for this specific kernel:

linux-tools-4.19.84-microsoft-standard

linux-cloud-tools-4.19.84-microsoft-standard

You may also want to install one of the following packages to keep up to date:

linux-tools-standard

linux-cloud-tools-standard

Barthol

In WSL2, perf can be used – Saharanpur

@Saharanpur Hi, I upgraded wsl1 to wsl2. However, it shows the semilar error message in wsl1: – Barthol

Did you install any linux*-tools* package(s) that you can see in aptitute or whatever package-manager you use? Your 2nd error message even lists some package names to install. – Obovate

@PeterCordes Hi, I searched for these packages but found nothing. I guess these names are generated automatically. I found the correct answer below. – Barthol

It didn't say that in your question so it seemed like there'd be some hope that the distro would package linux-tools since the kernel supports perf. But apparently not. Silly Microsoft :/ – Obovate

Answer 1

WARNING: perf not found for kernel 4.19.84-microsoft

Because WSL2 uses custom Linux kernel. Its source code can be found here microsoft/WSL2-Linux-Kernel. We have to compile perf tools from it.

Procedure

• Install required build packages. If you are using Ubuntu in WSL2 this is the required command:

sudo apt install build-essential flex bison libssl-dev libelf-dev

• Clone the WSL2 Linux kernel repository:

git clone --depth=1 https://github.com/microsoft/WSL2-Linux-Kernel.git

• Go to perf folder and compile it:

cd WSL2-Linux-Kernel/tools/perf
make

perf executable file will be in that folder.

Saharanpur

Perf tool sources have standalone version at mirrors.edge.kernel.org/pub/linux/kernel/tools/perf which is easier to use than doing full linux kernel git checkout or download. This site have perf for several recent versions of upstream linux kernel; they should work for any kernel. Or you can try to generate same targz file from WSL2 git with make perf-targz-src-pkg. Also: did you check the generated perf executable - what works and what does not; what is in perf list, what says perf stat for simple "echo 1", are there hardware events? – Stopover

@Stopover I didn't check every options but ./perf stat works as usual. – Saharanpur

Make sure you have WSL 2 installed other wise you will get perf_event_open(..., 0) failed unexpectedly with error 22 (Invalid argument) as described here: github.com/microsoft/WSL/issues/329#issuecomment-564454956 – Sofia

I'm getting this:- Error: Unable to find debugfs/tracefs Hint: Was your kernel compiled with debugfs/tracefs support? Hint: Is the debugfs/tracefs filesystem mounted? Hint: Try 'sudo mount -t debugfs nodev /sys/kernel/debug' rebroad@J3PMPJP:~/bin$ sudo mount -t debugfs nodev /sys/kernel/debug mount: /sys/kernel/debug: unknown filesystem type 'debugfs'. – Hizar

hello @Saharanpur , can you help me about an issue. I have done all of these, but i get this msg in the perf folder: WARNING: perf not found for kernel 5.10.16.3-microsoft – Roshelle

I succeed too run perf, but only generic event are available for me. No hardware events (like the number of AVX arithmetic operation or the amount of read/written MiB from/to RAM on Skylake). I used the version 5.10. Interestingly, perf top does not see Windows application. – Kedge

@JérômeRichard: When you say "generic events" worked, do you mean hardware PMU events like cycles and instructions, but not uarch-specific stuff like mem_inst_retired.any? Or do you only mean kernel software events like task-clock, page-faults, and context-switches, but not instructions? A recent answer installed a non-WLS2 version of perf and found no HW events were working. – Obovate

@PeterCordes I should have said system events rather than generic events. perf stat reports: task-clock, context-switches, cpu-migrations, page-faults. But the others like instructions or cycles are unsupported. – Kedge

@JérômeRichard: Ok, that makes more sense, I'd expect either no PMU events or all of them. With none being most likely since it runs in a VM, thus the hypervisor would need to virtualize the PMU for it to work. – Obovate

Answer 2

You can install linux-tools-generic.

apt install linux-tools-generic

Then run perf using the install path /usr/lib/linux-tools/<linux-version>-generic/perf.

Some tools, like flamegraph, will use environment variable PERF as the perf path.

PERF=/usr/lib/linux-tools/<linux-version>-generic/perf flamegraph -- my_program

Incite

Thanks for this answer. When I was typing perf in the cli it was using /usr/bin/perf instead of /usr/lib/linux-tools/5.4.0-104-generic/perf – Grisette

Answer 3

The accepted answer works. However, some features are missing.

In order to get useful and demangled information, I had to install the following libs and then run make again.

• libbabeltrace-dev
• libunwind-dev
• libdw-dev
• binutils-dev
• libiberty-dev

I'm not sure if all of them are necessary. However, those are adequate for cargo-flamegraph (my usecase) to work.

Hanging

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review – Pertinacity

This is actually a really good hint. – Janejanean

This is helpful. thanks. I also had to install libssl-dev to enable the libcrypto requirement. – Ringer

Here's a bigger list: gist.github.com/LoganDark/8f9ba939bbe4b73d057ceee46b07e20f – Imago

Answer 4

You could install some generic version of perf, rather than the WSL2 version, like:

sudo apt install linux-tools-5.4.0-126-generic linux-tools-common

And then when you run perf, it will error out, like:

$ perf
WARNING: perf not found for kernel 5.10.16.3-microsoft

  You may need to install the following packages for this specific kernel:
    linux-tools-5.10.16.3-microsoft-standard-WSL2
    linux-cloud-tools-5.10.16.3-microsoft-standard-WSL2

This is because script /usr/bin/perf always trying to get the perf binary from uname -r

$ grep uname `which perf`
full_version=`uname -r`

We could replace /usr/bin/perf with the actual perf :

mv /usr/bin/perf /usr/bin/perf.bk && ln -s /usr/lib/linux-tools/5.4.0-126-generic/perf /usr/bin/perf

and then:

$ perf stat ls 1>/dev/null

 Performance counter stats for 'ls':

              1.79 msec task-clock:u              #    0.827 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
               112      page-faults:u             #    0.063 M/sec
   <not supported>      cycles:u
   <not supported>      instructions:u
   <not supported>      branches:u
   <not supported>      branch-misses:u

       0.002158900 seconds time elapsed

       0.002182000 seconds user
       0.000000000 seconds sys

I think it is expected the hardware/cache counters are not available on WSL2

Outcome

WSL2 runs inside a VM. For HW PMU events to work, the VM + hypervisor would need to support it, so it doesn't let you profile stuff outside of guest VM. (And potentially even get last-branch-record tracing). Last I read, there was some work happening to make KVM support PMU access for Linux-on-Linux virtualization, but IDK if that ever got completed. And clearly WSL2 doesn't do it. – Obovate

This is exactly the kind of thing I get now on my machine too. Thank you for clarifying this long standing issue. This is sad most events do not work since it makes perf nearly useless in WSL... – Kedge

This is the only way I could get Hotspot to work on WSL2. – Lenten

Answer 5

If you follow the accepted answer, make sure you read the complains the make command prints at the start, as it might be missing some headers and disables functionality.

For me it disabled tui, gtk and demangling to name a few features.

Furculum

So which specific packages did you have to install to satisfy those optional dependencies and enable those features? That would make this answer more useful to future readers. – Obovate

Answer 6

I have experienced an error when trying to run perf stat using both the compiled and linux-tools-generic versions:

./perf stat -d ls
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (slots:u).
/bin/dmesg | grep -i perf may provide additional information.

Any rituals for updating/enabling/disable WSL2 or reinstalling Ubuntu image didn't fix it.
I have found an explanation in some newsletter:

Add [email protected] as this issue is in virtualization env.

The topdown metrics events became default since commit 42641d6f4d15 ("perf stat: Add Topdown metrics events as default events"). The perf will use 'slots' if the /sys/bus/event_source/devices/cpu/events/slots is available.

Unfortunately, the 'slots' may not be supported in the virualization environment. The hypervisor may not expose the 'slots' counter to the VM in cpuid. As a result, the kernel may disable topdown slots and metrics events in intel_pmu_init() if slots event is not in CPUID. E.g., both c->weight and c->idxmsk64 are set to 0.

There will be below error on Icelake VM since 'slots' is the leader:

$ perf stat Error: The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (slots). /bin/dmesg | grep -i perf may provide additional information.

This is because the stat_handle_error() returns COUNTER_FATAL when the 'slots' is used as leader of events.

There are three options to fix the issue.

1. Do not expose /sys/bus/event_source/devices/cpu/events/slots to userspace so that pmu_have_event(pmu->name, "slots") returns false.

2. Run cpuid at perf userspace and avoid using 'slots' if it is not supported in cpuid.

3. Do not fatal perf if the leader is failed. Do not create events for an evsel if its leader is already failed.

So I tried to manually specify the events, not specifying slots:

./perf stat -e cache-misses -d ls

 Performance counter stats for 'ls':

             20145      cache-misses
            555870      L1-dcache-loads
             25832      L1-dcache-load-misses     #    4.65% of all L1-dcache accesses
   <not supported>      LLC-loads
   <not supported>      LLC-load-misses

       0.000948558 seconds time elapsed

       0.001025000 seconds user
       0.000000000 seconds sys

So, perf is finally useable in my WSL. Hope this helps people with similar errors.

Ency

Answer 7

In my case (WSL2 Debian), I fixed this by installing linux-perf. It was after trying a couple answers here and reading this WSL Issue.

Oberon