How to make AddressSanitizer not stop after one error (and other issues)

Question

I'm running OS X, 10.8.5; I've installed llvm 3.4 via homebrew (clang version 3.4 (tags/RELEASE_34/final) ), and I'm building with -fsanitize=address. I can get asan working with simple demo programs, but when building against our codebase, I have several problems (although I'd really just like an answer to #1):

1. 3rd party libraries are generating asan errors, and asan is terminating my app on the first occurrence. I would think that there would be some kind of (runtime/compile-time) option to tell asan to keep going after finding an error. Specifically, I see this:

bash-3.2$ ASAN_SYMBOLIZER_PATH=/usr/local/Cellar/llvm34/3.4/lib/llvm-3.4/bin/llvm-symbolizer ./unit_test
Start testing of PathTrieTest
Config: Using QTest library 4.8.2, Qt 4.8.2
PASS   : PathTrieTest::initTestCase()
PASS   : PathTrieTest::pathTrieNodeTest()

=================================================================
==76647==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61600019e588 at pc 0x10891ddd3 bp 0x11312ba90 sp 0x11312ba58
WRITE of size 48830 at 0x61600019e588 thread T3
    #0 0x10891ddd2 in wrap_readdir_r (/usr/local/lib/llvm-3.4/lib/clang/3.4/lib/darwin/libclang_rt.asan_osx_dynamic.dylib+0x11dd2)
    #1 0x10ac23571 in QFileSystemIterator::advance(QFileSystemEntry&, QFileSystemMetaData&) (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xef571)
    #2 0x10abd86d3 in QDirIteratorPrivate::advance() (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xa46d3)
    #3 0x10abd7a7f in QDirIteratorPrivate::QDirIteratorPrivate(QFileSystemEntry const&, QStringList const&, QFlags<QDir::Filter>, QFlags<QDirIterator::IteratorFlag>, bool) (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xa3a7f)
    #4 0x10abd8b68 in QDirIterator::QDirIterator(QDir const&, QFlags<QDirIterator::IteratorFlag>) (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xa4b68)
    #5 0x10abd7609 in QDirPrivate::initFileLists(QDir const&) const (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xa3609)
    #6 0x10abd5394 in QDir::count() const (/Volumes/ToolChain/qt-4.8/lib/QtCore.framework/Versions/4/QtCore+0xa1394)
    #7 0x1084c205d in get_count(QFileInfo&) /Users/stebro/dev_vm/ui/ui/fsinfoprovider.cpp:36
...

This error doesn't cause the app to terminate when it's running unsanitized.

2. I can't get code to link when using the -fsanitize=undefined (or -fsanitize=address,undefined) options. I am including the -fsanitize=undefined line on both my compile & link commands, but I get link errors such as:

      Undefined symbols for architecture x86_64:
        "typeinfo for __cxxabiv1::__class_type_info", referenced from:
            __ubsan::checkDynamicType(void*, void*, unsigned long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
            isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
            findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
      "typeinfo for __cxxabiv1::__si_class_type_info", referenced from:
            isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
            findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
      "typeinfo for __cxxabiv1::__vmi_class_type_info", referenced from:
            isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
            findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in libclang_rt.ubsan_osx.a(ubsan_type_hash.o)
   

3. I can't get blacklists to work, and -mllvm -asan-globals=0 or -mllvm -asan-stack=0 doesn't seem to work like I'd expect. For example, the latter don't suppress the generation of the error listed in #1 above, and creating a blacklist that looks like the one below doesn't suppress the errors either:

   fun:QDirPrivate::initFileLists fun:get_count fun:opendir2

4. Finally, executables generated with these asan options cause lldb to crash. I'm using the lldb shipped with XCode 5 tools; there was no lldb deployed with the homebew llvm package, and I can't figure out how to build it. The build instructions enter link description here have a dead link that points to the source you're supposed to use; pulling the source directly from the svn repository, using:

svn co http://llvm.org/svn/llvm-project/lldb/tags/RELEASE_34/final lldb

results in code that doesn't compile (errors provided upon request).

Assault

I found a way to work around the Qt heap overflow you're seeing. I posted it as an answer below. – Chen

Answer 1

You can build your project with -fsanitize=address -fsanitize-recover=address flags, and run with environment variable ASAN_OPTIONS=halt_on_error=0. Source: https://github.com/google/sanitizers/wiki/AddressSanitizer

Aramenta

This seems to be a new feature added since I asked the question. Thanks for the update. – Assault

ASAN_OPTIONS is not working in windows , I tried the above example ,when I give ASAN _OPTIONS it gives unrecognized option for it. It works fine with ubuntu. – Escadrille

how to " run with environment variable ASAN_OPTIONS=halt_on_error=0" at MAC? – Naseby

Seems to be ignored at least on macos. Neither this nor disabling specific kind of errors, nor a suppression file seem to get respected. Anyone know how to use ASAN on macos without xcode? – Diluvium

This only seems to print first error. Any way to make it print multiple ASAN errors? – Rebak

Answer 2

1. Addressability bugs (especially memory corruptions like the OOB write you've mentioned) are usually severe enough to worry about. Ignoring them can't undo their effects, and the program is very likely to crash in a totally unrelated place after you've skipped a heap corruption. You can try to build your own Clang with checks in readdir_r disabled, but I'd try to fix the error in the first place.
2. Can you please file a bug with reproduction steps at http://code.google.com/p/address-sanitizer ?
3. Blacklists only disable addressability checks in the blacklisted functions by not instrumenting them, they do not check the stacks for certain frames. Your particular check occurs in a function interceptor compiled into the runtime library, so you can't even blacklist that. If the error in #1 is in some system library and can't be easily worked around we can look into adding a runtime option to disable the readdir_r checks.
4. Again, a bug report will be appreciated.

Mcmahon

There's a legitimate use case for disabling halting on AddressSanitizer errors: I want to see how many AddressSanitizer the whole program generates, not just if it has 0 or >= 1 errors. – Rebak

Answer 3

By the way -- I found a way to work around this heap overflow in Qt if you're building it from source. I believe the problem is related to 32-bit / 64-bit inodes (_DARWIN_FEATURE_64_BIT_INODE is not defined when I build Qt for some reason).

diff --git a/src/corelib/io/qfilesystemiterator_unix.cpp b/src/corelib/io/qfilesystemiterator_unix.cpp
index 029b989..76b176f 100644
--- a/src/corelib/io/qfilesystemiterator_unix.cpp
+++ b/src/corelib/io/qfilesystemiterator_unix.cpp
@@ -75,6 +75,7 @@ QFileSystemIterator::QFileSystemIterator(const QFileSystemEntry &entry, QDir::Fi
         size_t maxPathName = ::pathconf(nativePath.constData(), _PC_NAME_MAX);
         if (maxPathName == size_t(-1))
             maxPathName = FILENAME_MAX;
+        maxPathName = 99999;
         maxPathName += sizeof(QT_DIRENT) + 1;

         QT_DIRENT *p = reinterpret_cast(::malloc(maxPathName));

Chen

Thanks. We've since moved to Qt5 and the problem seems to have disappeared. – Assault

Answer 4

On Windows you can just set ASAN_OPTIONS=continue_on_error=1 and unique full error reports will go to stdout along with a summary upon exit(). No need to compile with a different compiler flag. Consider this blog post.

Wira