Memory/Address Sanitizer vs Valgrind

Question

I want some tool to diagnose use-after-free bugs and uninitialized bugs. I am considering Sanitizer(Memory and/or Address) and Valgrind. But I have very little idea about their advantages and disadvantages. Can anyone tell the main features, differences and pros/cons of Sanitizer and Valgrind?

Edit: I found some of comparisons like: Valgrind uses DBI(dynamic binary instrumentation) and Sanitizer uses CTI(compile-time instrumentation). Valgrind makes the program much slower(20x) whether Sanitizer runs much faster than Valgrind(2x). If anyone can give me some more important points to consider, it will be a great help.

Sugarcoat

Answer 1

I think you'll find this wiki useful.

TLDR main advantages of sanitizers are

• much smaller CPU overheads (Lsan is practically free, UBsan/Isan is 1.25x, Asan and Msan are 2-4x for computationally intensive tasks and 1.05-1.1x for GUIs, Tsan is 5-15x)
• wider class of detected errors (stack and global overflows, use-after-return/scope)
• full support of multi-threaded apps (Valgrind support for multi-threading is a joke)
• much smaller memory overhead (up to 2x for Asan, up to 3x for Msan, up to 10x for Tsan which is way better than Valgrind)

Disadvantages are

• more complicated integration (you need to teach your build system to understand Asan and sometimes work around limitations/bugs in Asan itself, you also need to use relatively recent compiler)
• MemorySanitizer is not easily usable at the moment as it requires rebuilding all dependencies under Msan (including all standard libraries e.g. libc++); this means that casual users can only use Valgrind for detecting uninitialized errors
• sanitizers typically can not be combined with each other (the only supported combination is Asan+UBsan+Lsan) which means that you'll have to do separate QA runs to catch all types of bugs

Handyman

Note that valgrind supports multi-threaded apps properly. However, only one thread executes at the same time (so effectively, an application running under valgrind runs like if it was on a single core) – Langer

@Langer Yeah, that's what I meant, they are effectively single-threaded. In many cases this is a performance killer. – Handyman

About the thread safety in Sanitizer, I found ThreadSanitizer : clang.llvm.org/docs/ThreadSanitizer.html . Is it good enough to handle multi-threaded execution? Can I use ThreadSanitizer with other sanitizers like: MemorySanitizer? – Sugarcoat

@Sugarcoat Regarding TSan - yes, all sanitizers fully support multi-threading. Note however that TSan's overhead is much higher (8x CPU) and it only works on x86_64 Linux. – Handyman

@Sugarcoat Regarding combining sanitizers - the only combination that's supported is ASan+UBSan+LSan. Usually people run their software under each sanitizer independently. This is important disadvantage, I'll add it to the list. – Handyman

You may want to move "overhead point" to advantages, since we're comparing to Valgrind. Meanwhile, TIL a disadvantage of leak sanitizer that you might add to your answer. Valgrind tracks pointers to memory, leak sanitizer doesn't. If you SIGTERM/SIGINT debuggee, sanitizer won't print leaks. Valgrind however will do so and the fact that upon signal a bunch of memory won't get freed won't clutter the report because Valgrind by default won't consider as a leak a memory to which a pointer is found. – Eugenaeugene

@Eugenaeugene Thanks, I moved the overhead point, it indeed makes more sense as advantage. "Valgrind tracks pointers to memory, leak sanitizer doesn't" - why do you think so? I'm adamant that Lsan does track pointers and uses it to not report leaks for reachable pointers. – Handyman

@Handyman hmm, weird… For some reason I can't reproduce anymore Valgrind not reporting memory leak when there's a pointer remains… Have I imagined it…? Anyway, to answer your question: this answer demonstrates Lsan reporting memory leaked despite a pointer still in scope. You'd probably say that memory indeed leaked there: well, to make more obvious that no pointer tracking happens here append right after the while (argc - 1); line a free(a) call. Now the app no longer leaks and you gonna get a false-positive. – Eugenaeugene

@Eugenaeugene The function at this link returns when signal arrives so malloced pointer remains in upper frame and is not considered to be reachable when Lsan analysis is triggered (after return from main). So I'd say Lsan behaves correctly in this case. – Handyman

thread sanitizer works also on Mac OSX as of clang 12.0.0 clang.llvm.org/docs/ThreadSanitizer.html – Singlehearted

@Singlehearted right but the question target memory-related sanitizers. – Handyman

Answer 2

One big difference is that the LLVM-included memory and thread sanitizers implicitly map huge swathes of address space (e.g., by calling mmap(X, Y, 0, MAP_NORESERVE|MAP_ANONYMOUS|MAP_FIXED|MAP_PRIVATE, -1, 0) across terabytes of address space in the x86_64 environment). Even though they don't necessarily allocate that memory, the mapping can play havoc with restrictive environments (e.g., ones with reasonable settings for ulimit values).

Ralli

"Havoc" might be too strong. App would just abort at start with easily googleable error message suggesting user to reset ulimit. – Handyman

@Handyman changing ulimit is sometimes easier said than done (especially in a distributed test environment that is not under your direct control); also, there are other (perhaps pathological cases) where one wants to run the sanitizer-instrumented binary under, say, valgrind, where the test harness sees references to this otherwise unused space as an actual use (and thus needs to allocate accounting space for it). Not to mention that it's also disconcerting/confusing to see top and ps show a process size of 124.2t... – Ralli

Well, distributed test environment will always be non-trivial for Valgrind or any other dynamic tool (yes, I've tried them all) so I'm not sure how sanitizers are worse than others. Sanitized apps can not be run under valgrind (there's some memory conflict). – Handyman