Why is raising plackup (or starman) memory usage?

Question

I have this simple PSGI application (app.psgi).

use strict;
use warnings;

my $app = sub {
    my $mem = `ps -o rss= -p $$`;
    $mem =~ s/^\s*|\s*$//gs;
    return [ 200, [ 'Content-Type' => 'text/text' ], [ $mem ]];
};

I was requested the above 1000 times and got increased memory usage. Depending on how was started the server, got:

• plackup - memory usage is raising at first 3 requests and remain constant for the next 997 requests

• plackup -r - memory usage is randomly raising (not at every request) by 4k.

• starman - like above, the memory usage is randomly raising by 4k, but with slower rate

The question is:

• WHY is raising the memory usage ? Where is the leak, and how achieve constant memory usage (especially on starman), because i don't want run out of memory at long run. (OK, it is possible define for example --max-requests 100), but it is not an answer for the memory usage.
• or - what is wrong in my example?

If anyone want test this too - here is my script for the fetching:

use strict;
use warnings;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => 'http://localhost:5000');

my $old_mem = 0;
print "req#\tmem\n";
foreach my $i (1..1000) {
    my $res = $ua->request($req);
    (my $mem = $res->content) =~ s/\D//g;
    next if( $mem == $old_mem );
    print "$i\t$mem\n";
    $old_mem = $mem;
}

My results:

plackup                 plackup -r              starman
req#    mem             req#    mem             req#    mem
1       7780            1       3924            1       3280
2       7800            2       4296            5       3728
3       7804            3       4304            8       3280
                        ...                     ...
                        ... deleted             ... deleted
                        ...                     ...
                        839     4596            994     3912
                        866     4600            998     3908
                        962     4604            1000    3912

So,

• why plackup raising in first 3 requests?
• plackup -r - 4k increase (see last lines) - at the beginning much more
• starman - raising too, but with default 5 workers in slower rate (3280->3912)

Versions:

# cpanm Plack Starman
Plack is up to date. (0.9979)
Starman is up to date. (0.2010)
# perl -v

This is perl 5, version 12, subversion 3 (v5.12.3) built for darwin-thread-multi-2level

Harelda

"plackup - memory usage is raising at first 3 requests and remain constant for the next 997 requests" That means some modules are lazy-loaded in the first few requests. After that there's no leak. – Gelhar

Starman by default enables keep-alive and HTTP pipelining, meaning if you send 1000 requests in the short period of time you'll have these connections connected, unless you explicitly disconnect them. I can confirm this using ApacheBench - the memory temporarily increases, but when they disconnect/timeout, the memory gets down to where it was. – Gelhar

wow. Thank you for the explanation. :) I was afraid that I have a problem with my perl or so. – Harelda

I confirm I see the memory increase in the Restarter case (-r option) - I'll see if there's any place that is leaking. Because -r is usually used during the development I wouldn't consider it a huge deal - but it's always nice to have no leaks :) – Gelhar

Answer 1

Based on miyagava's comments the answer is:

"plackup - memory usage is raising at first 3 requests and remain constant for the next 997 requests" That means some modules are lazy-loaded in the first few requests. After that there's no leak. – miyagawa 14 hours ago

Starman by default enables keep-alive and HTTP pipelining, meaning if you send 1000 requests in the short period of time you'll have these connections connected, unless you explicitly disconnect them. I can confirm this using ApacheBench - the memory temporarily increases, but when they disconnect/timeout, the memory gets down to where it was. – miyagawa 14 hours ago

thanx.

Harelda

Answer 2

Did you use the latest version? I cannot reproduce your output.

With "plackup":

sidburn@sid:~/perl/plack$ ./memory.pl 
req#    mem
1   5340
2   5380

With "plackup -r":

sidburn@sid:~/perl/plack$ ./memory.pl 
req#    mem
1   4860
2   5060

With "starman":

sidburn@sid:~/perl/plack$ ./memory.pl 
req#    mem
1   5176
5   5224
6   5176
7   5224

Versions:
Perl: 5.12.1 & 5.12.3
Plack: 0.9979
Starman: 0.2010

Bioplasm

Yes: Plack is up to date. (0.9979) and starman was only 0.2008, but after the upgrade the result is the same... – Harelda

Is your Perl reasonably up-to-date? – Vivienviviene

havent 5.14 yet. This is perl 5, version 12, subversion 3 (v5.12.3) – Harelda

It also works with Perl 5.12.3. On which system do you use it? Linux? Which? A special distribution? I used Debian Squeeze with perlbrew. – Bioplasm

Also check the value of usemymalloc in your perl Config (i.e. perl -V | grep usemymalloc). If it says y, perl uses its own malloc, and doesn't return memory back to the operating system once it's allocated, which may not be the behavior your want to see in the persistent environment such as web servers. See INSTALL file for details. – Gelhar