How to debug memory corruption on Linux

Question

I am faced with a very smart memory corruption in my C application.

Corruption occures on high load.

So I tried purify, valgrind, mprotected and also I tried to write my own simple protection mechanism.

Purify / Valgrind - doesn't help because it reduces performance of my application and problem was not reproduced.

mprotected usage just move corruption to other memory location. (because it required to align memory to page size).

My simple protection mechanism doesn't work because it also reduces a performance.

How I can debug my application without performance degradation?

Engelbert

I'd be surprised if Valgrind wouldn't find it.... – Traps

Corruption occures on hight load. How do you know? Also: monitor the process's VSIZ+RSS. Could be that you are just leaking memory. – Collenecollet

How does the memory corruption manifest itself? Is your application single-threaded or multi-threaded? Is your hardware stable under stress tests? – Minacious

Answer 1

use sanitizer:

add -fsanitize=address or -fsanitize=thread on your compilation flags and it will likely point you the defects.

You may also add -O0 for removing optimization (better backtraces) and -g to keep symbols/debug info in your binary.

Devault

Answer 2

If you have 64 bit you can use a custom malloc() that always does mmap() and a custom free() that does munmap() and another mmap() on the same memory. Protect this stuff with a mutex to avoid a deadly race condition. This changes the behavior to fault on the first access to freed memory.

If it doesn't find it, adjust the custom malloc() to move the allocated buffer as high as possible in the mmapped area.

Note you can't do this in 32 bits because this burns address space like crazy.

Retinoscopy

Answer 3

Purify / Valgrind - doesn't help because it reduces performance of my application and problem was not reproduced.

While reading that, I believe you have not only a memory corruption but also one or more race conditions.

So I would give you the hind to first run with helgrind to find the race conditions. But helgrind is not aware of memory ordering if you use std::atomic. In this case it reports false positives and is more or less unusable. For that situation I don't know any tool which checks for memory ordering, which is a big problem in the moment.

How can I debug my application without performance degradation?

The question is: Why your failure depends on performance? Do you have I/O in parallel or running multiple tasks/threads? If so, reduce the speed of that tasks or threads or the I/O also, maybe you can force the error to raise.

Hints for slow down other threads/tasks: On linux you can bind a thread/task to a cpu/core and you hopefully can consume more power on that single core by adding multiple "stopper" tasks also on that core. taskset. You may also compile special parts of your code with -O0 or other hacks.

I know it is quite a nightmare to find bugs which are gone if you have debugging tools in place. But we can not really help, as we have nothing to take a look on... so it is a bit reading the crystal ball!

Wilberwilberforce

This isn't C++ -- there is no std::atomic. – Wot

@JL2210: I have no idea why I saw this question as it was from 5 years ago. Maybe it is now with c++ :-) And the problem is generic for c++ and c. Anyone who find that much to late answer useful may also find the hint to atomic useful. But yes, c has no atomics :-) – Wilberwilberforce

Joshua (one of the other answerers) bumped it to "Active" by answering it. – Wot

This has never been tagged [c++], though. – Wot