Pthreads - High memory usage

Question

I am programming something in C that creates a lot of Pthreads in Linux on a 256Mb system. I usually have +200Mb free.

When I run the program with a low amount of threads it works, but once I make it create around 100 threads it gives errors because the system runs out of memory. I did several tests and each threads use almost 2Mb. The stack size of the threads is set to 16Kb.

The code I use to create each thread:

pthread_attr_t attr;
pthread_attr_init(&attr);
size_t stacksize;
stacksize = (double) 16*1024;
int res = pthread_attr_setstacksize (&attr, stacksize);
int res2 = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
if (res != 0 || res2 != 0) {
    logs << "pthread_attr_XX: error "+int2string(res);
    exit(-1);
}
pthread_t id;
pthread_create(&id, &attr, &Class::thread_callback, &some_var);

Is it normal or am I missing something? Thanks.

Demetrius

Are you certain that your system is running out of RAM, or is it possible that you're running out of other resources (thread descriptors, perhaps)? – Ayers

@mah: thread descriptors would be very unlikely in Linux, as under the hood they are handled just like full process for most things and take from the process id pool, until that is exhausted. More likely is that it is memory problems are cased by things being created on the heap and exhasting heap space. – Veracity

What are the threads doing? You might want to rethink a design that spawns 100 threads on a machine with such limited memory resources. Maybe look at select(). – Rifling

@mah, yes, i can see in other terminal that used memory increase until it runs out. – Demetrius

Is there a TLS used (arrays with __thread specifier)? This can come from libraries too. Do your threads use a lot memory from malloc? Is it really no error returned in res? Your value is exactly the biggest prohibited value according to line 74 of this file. The check is "/* Catch invalid sizes. */ \n if (stacksize < PTHREAD_STACK_MIN) \n return EINVAL;" – Tepid

NeDark, Can you do an strace -f or ltrace -S -f of your program and post results online? Any memory allocation will be visible in such traces. – Tepid

Have you tried a test program that only creates 100 threads, but does nothing, for comparison to your program? Any chance you have a memory leak instead? – Sporophore

Answer 1

Not sure it will help, but try calling setrlimit with RLIMIT_STACK to limit the stack size to 16k before creating your first thread.

Mahratta

16 kb may be too small, so ulimit -s 40 and ulimit -s 70 should be tried too. – Tepid

@osgx: the commands you suggested did the trick, thanks. I didn't try with setrlimit, but I guess it'll work too since it's based on the same. Now I can open a lot of threads having a low memory usage. It looks that pthread_attr_setstacksize wasn't working as expected and a change on the global limit was needed – Demetrius

Answer 2

The system threads library is just not designed to support large numbers of threads on systems with very limited memory. You need to either use a threading library designed for that purpose or, better, use fewer threads.

Heronry

Answer 3

Yeah, it's fairly common for pretty much any operating system to choke and die on that many threads. It's not a problem that OS makers (Linux or otherwise) have given much attention to since very few systems have that many CPUs, so your code wouldn't likely executing much faster with 100 threads than it would with 8 (or however many CPUs you have).

I think what you need to do is some sort of thread pooling. I also suspect the ulimit answer on this page would be useful if you really need that many threads, but like I said, I don't think that many threads gets you much in most circumstances and if you fix it in your program instead of in the system, it'll make it more portable.

Adelina

Answer 4

Why stacksize = (double) 16*1024; ? It is integer.

Try to set it to 32 or 64 kilobytes, because 16*1024 may be not allowed; there can be also TLS in the stack (even you doesn't use TLS; libraries do, including libc).

So, change the line to

stacksize = 64*1024; 

and check how much memory is consumed.

Tepid

Answer 5

If you want lower overheads consider user-space threading technologies such as fibers, ala co-operative task management.

http://en.wikipedia.org/wiki/Fiber_(computer_science)

http://www.evanjones.ca/software/threading.html

GNU Portable threads:

http://www.gnu.org/software/pth/

Boost C++ co-routines:

http://www.boost.org/doc/libs/1_60_0/libs/coroutine/doc/html/index.html

Windows-only fibers as an FYI:

http://msdn.microsoft.com/en-us/library/ms682661(v=vs.85).aspx

See the Wikipedia link for more example implementations.

Overthrow

Answer 6

May be this is the reason:

"(Recognizing leaks

If you create a joinable thread but forget to join it, its resources or private memory are always kept in the process space and never reclaimed. Always join the joinable threads; by not joining them, you risk serious memory leaks.)"

Read it here: http://www.ibm.com/developerworks/library/l-memory-leaks/

Gillian

I don't think the problem is that the OP isn't joining the threads. The problem is related to the amount of memory used when creating the threads. – Wingfield