Perl: How to free memory allocated for a scalar without access to the Perl variable?

Question

This question is related to an answer to a former question about memory handling by Perl. I've learned that one can free memory in Perl by explicitly using the undef function on an available scalar and using Devel::Peek or Devel::Size or such one can see how many memory is allocated for a scalar. In all those cases the scalars debugged are used within their scope.

But is it possible to debug things like allocated memory outside the scope of variables, just on the level of a Perl interpreter? Something like searching for all allocated memory for all "things" that are a scalar in the current interpreter and print their associated data, like current value or such?

And if that's the case, if one does already have that information, is one even able to free the known memory? Just like calling undef on a scalar, but without the scalar, something more low level, like on those "things" output of Devel::Peek.

What I'm thinking about is having a mod_perl cleanup handler executed after a request, scanning the current mod_perl interpreter for large chunks of data and freeing them manually. Simply because I decide that large blocks of allocated data are of no use anymore, even if Perl thinks otherwise:

Finally and perhaps the biggest win is memory re-use: as calls are made into Perl subroutines, memory allocations are made for variables when they are used for the first time. Subsequent use of variables may allocate more memory, e.g. if a scalar variable needs to hold a longer string than it did before, or an array has new elements added. As an optimization, Perl hangs onto these allocations, even though their values "go out of scope".

https://perl.apache.org/docs/2.0/user/intro/overview.html#Threads_Support

I could find a lot of monitoring and debugging packages around low level memory access, but no hint yet how one could call something like the undef function on some low level Perl struct in Perl. Might simply not be possible without any XS or such...

Hg

The "low level Perl struct" that holds the memory is the var itself, so undef is indeed the right tool. The variables themselves are found in a function's pad. You can use PadWalker's peek_sub to get references to the vars of a sub from a sub reference, and you can get references to the named subs from the namespace (%::). – Appurtenant

Are you unsure that your approach works? Because it sounds like an answer I would like to accept. – Veracruz

I just don't have the time to write it up nicely. – Appurtenant

Would be "nice enough" for me, but I can wait... ;-) – Veracruz

@Appurtenant are you contributing to the core? – Titty

@simbabque, I did small thing here and there, that's all. (A fix to the 5 functions that return lvalues, some fixes to substr indexing, stopped splice(@a, 5) from warning if @a has fewer than 5 elements, a couple of other things I can't remember, inclding some doc fixes.) – Appurtenant

If your intention is to reduce the memory that the perl process is using, I don't think it is possible to make perl release memory back to the OS. I remember struggling with that on Windows a few years back. Running undef will free up some space for perl to reuse memory, but I also recall seeing some memory leaks when I tested this. I had tested by creating large arrays then undefing them and watched as they continued to consume more memory. I don't think I have this code anymore, but I can try to recreate if you are interested. – Nitrogenous

@NathanLoyer I am interested and would be great if you could let us have a look. – Ferdinana

Fact: Only Chuck Norris can make a running Perl program give memory back to the OS. – Ginder

@Appurtenant I just came across this question again when looking for something to answer. If you have some time to spare, would you mind writing up the answer you hinted? – Titty

@sumbabque, No, way too much work. What I posted was really just a starting point. And it overlooked a major issue: Which variable is safe to undef? You can't clear variables in subs in your call stack. Maybe caller can help? – Appurtenant

@simbabque, I've posted an answer. – Appurtenant

@Appurtenant great, thank you. Good stuff. :-) – Titty

Answer 1

is it possible to debug things like allocated memory outside the scope of variables

There really isn't any such memory. Any memory allocated outside of variables is surely needed. As you yourself point out, it's the memory allocated for variables that make up most "wasted" space.

but no hint yet how one could call something like the undef function on some low level Perl struct in Perl.

It's because there are no such structs.

Just like calling undef on a scalar, but without the scalar, something more low level, like on those "things" output of Devel::Peek.

Devel::Peek's only function, Dump, outputs things in variables. Like you've said, undef is what you'd want to clear these.

---

From the above, it's obvious you want to know how to free the memory associated with the variables in subs.

You also overlooked the fact that many operators have an associated variable (called "target") in which they return their result.

Approach 1

A simple way to clear all those variables would be to selectively clear the symbol table (%::). This would effectively "unload" every module. Be sure not clear core components (perl -E'say for sort keys %::'). And don't forget to clear %INC so the modules can be reloaded.

If clearing the symbol table is the approach you want to take, it might be less risky and time-consuming to take a snapshot of %:: early on, and restore that snapshot when it's time to clear the symbol.

Approach 2

If you didn't want to reload the modules, you could take attempt to locate every sub, and undef their vars, then undef the vars of their ops.

A sub's vars exists within its pads. Conveniently, so do opcode targets. There's a pad for each level of recursion the sub has experienced.

Given a reference to a sub, you can find the variables in a sub's pads. You can refer to PadWalker for an example of how to do this. You can't actually use PadWalker since it only returns one variable per variable name, even if there are more than one (due to more than one variable being declared with the same name, or due to recursion).

Captured variables and our variables should be left untouched. It's possible to detect whether a pad entry is one of these. (Again, refer to PadWalker.)

(Obviously, you could also looking into freeing the sub's extra pads!)

How do you find all the subs? Well, navigating the symbol table will give you most of them. Finding anon ones will be trickier.

Approach 3

The most efficient approach is to simply terminate the mod_perl thread/process. A new clean one will automatically be spawned. It's also the simplest to implement, as it's simply a configuration change (setting MaxRequestsPerChild to 1).

---

Another form of wasted memory is a memory leak. That's another large question, so I'm not touching it.

Appurtenant

Approach 3 sounds interesting, one would "only" need to take care about not loosing mod_perl benefits at all by loading all or at least most used packages at server startup, to feed the parent Perl interpreter with. perl.apache.org/docs/2.0/user/intro/… perl.apache.org/docs/2.0/user/handlers/… perl.apache.org/docs/2.0/user/handlers/… – Veracruz

Also, the child-replacement happens on request end, not on request start, so you even if startup takes a little time, it might not matter at all. – Appurtenant

@ikegami: Your answers full and interesting. Here is your award =) – Subinfeudation

Answer 2

I think you are looking for this answer to a similar question. Everything you really need to know you can find in the internals to the Devel::MAT::* submodules. Namely the Devel::MAT::Dumper.xs, which has the structure of the heap for perl interpreter. The module is designed to dump the heap at signal and analyze it later, but I think you can turn it into a runtime check. If you need help on reading the XS, look here.

Idiocy

Answer 3

To debug memory allocation you should recompile perl with -Accflags=-DPERL_MEM_LOG DOC

(see related question about how to recompile perl)

You maybe will be interested in MEMORY DEBUGGERS

To free perl scalar, just like when it leave her scope:

{ # Scope enter
     my $x; # Memory allocation
} # << HERE $x is freed

You should just reduce its variable REFCNT to zero by SvREFCNT_dec macro DOC

To free an SV that you've created, call SvREFCNT_dec(SV*) . Normally this call is not necessary (see Reference Counts and Mortality).

Here is pseudo code:

{ 
    $x;
    call_xs_sub( $x ); # << HERE $x is freed        
}

XS pseudo code:

call_xs_sub( SV *sv ) {
    ...
    SvREFCNT_dec( sv ); # <<HERE scalar is freed
    ...
}

To spy every memory allocation you should walk perl arenas.

At compile time you may view every place where variable is declared and accessed with help of B::Xref module

Or run perl with -Dm option (The perl should be compiled with corresponding options. See this topic):

perl -Dm script.pl

Subinfeudation

Re "To free perl scalar, just like when it leave her scope", As mentioned in the OP, Perl variables AREN'T freed when you leave their scope. Their REFCNT AREN'T decremented to zero. The OP is asking how to undef those variables (which would free their string buffers, array buffers, etc). undef (Perl) or sv_clear (XS) would be the appropriate way of doing that, not SvREFCNT_dec. – Appurtenant

...which leads to the bigger question of finding the variable. If you have a ref to the sub to which which they belong, you can use peek_sub to get the lexicals. – Appurtenant

...which leads to two larger questions: How do you get a list of subs (including anonymous subs), and how do you know if a var is safe to clear (variable of a running subs and a captured variables would not be safe to clear). This is what the OP needs answered. – Appurtenant

Oops, sv_clear is not undef. There's no XS equivalent to undef. – Appurtenant