Can ETW (event tracing for windows) be used to gather also memory statistics?

Question

Is it possible using ETW to also get memory statistics of all the processes and the system ? With memory statistics I mean : e.g. Commited bytes, private bytes,paged pool,working set,...

I cannot find anything about using xperf to get and see memory statistics. It is always about CPU , disk , network.

One could probably use performance counters to get that kind of information, but how can one overlay the statistics graphically in one chart (how to correlate/sync the timestamps) ?

Headspring

Answer 1

Your best bet on Windows 8.1 and higher is the Microsoft-Windows-Kernel-Memory provider, which records per-process memory information every 0.5 s. See https://github.com/google/UIforETW/issues/80 for details. UIforETW enables this by default when it is available.

You could also try the MEMINFO provider. It gives a system-wide overview of memory pressure. It shows the Active List (currently in use memory), the Standby List ('useful' pages not currently in use, such as the disk cache), and the Zero and Free lists (genuinely free memory). This at least lets you tell whether a system is running out of memory.

You could also try MEMINFO_WS and CONTMEMGEN but these are undocumented so I really don't know what they do. They show up in xperf -providers k but when I record with them I can't see any new graphs appearing. Apparently Microsoft ships these providers but no way to view them. Sigh...

If you want more memory details on Windows 7 -- such as per-process working sets -- your best bet is to have a process running which periodically queries this data and emits it in custom ETW events. This is available in a prepackaged form in UIforETW which can query the working set of a specified set of processes once a second. See the announcement post for how to get UIforETW: https://randomascii.wordpress.com/2015/04/14/uiforetw-windows-performance-made-easier/

UIforETW's Windows 7 working set data shows up in Generic Events under Task Name == WorkingSet. On Windows 8.1 the OS working set data (more detailed, more efficiently recorded) shows up under Memory-> Virtual Memory Snapshots.

Gine

Answer 2

You can trace memory usage with ReferenceSet kernel group. It includes the following traceflags:

PROC_THREAD+LOADER+HARD_FAULTS+MEMORY+FOOTPRINT+VIRT_ALLOC+MEMINFO+VAMAP+SESSION+REFSET+MEMINFO_WS

MEMORY = Memory tracing

FOOTPRINT+REFSET = Support footprint analysis

MEMINFO = Memory List Info (active, standby and oters you see from ResMon)

VIRT_ALLOC = Virtual allocation reserve and release

VAMAP = mapped files information

MEMINFO_WS = Working set Info

As you can see xperf can capture a lot of memory data when you sue the right flags.

Photovoltaic

Yep, it can capture a lot of data, but how do we get it to display that data, or otherwise extract it? In particular I can't find any way to get the MEMINFO_WS data displayed. In the Memory section of WPA I can see Memory Utilization (Active/Modified/etc. lists), Hard Faults, VirtualAlloc Comit Lifetimes, Pool Graphs, and Resident Set. Resident Set looks the most useful, but it seems to be displaying REFSET info, so what displays MEMINFO_WS data? – Gine

Resident Set only seems to grab a snapshot at the end of the trace, so it is poor for noticing trends over the course of a trace. I think I/we really need MEMINFO_WS data. – Gine