Advice needed for preventing file descriptor leaks in Raku

Question

Background

A while ago I wrote a Raku module to test out some ideas I had for making external process pipelines (e.g., grep raku | wc -l). I followed the traditional way of creating Pipes and setting up I/O redirections for the external processes.

Since there's no fork available in Raku, nor is it recommended due to threads in MoarVM, I resorted to using Proc::Async to start a wrapper process for each external command in a pipeline, and the wrapper would then sets up I/O redirections to connect the command processes and the pipes, and then close other FDs before exec-ing the real command.

Problem

This all seems to work well except there's still the possibility that other parts of the program using the module might start a sub process (e.g., via run, shell, Proc, or Proc::Async) in another thread after the pipe FDs are created but before they are closed, thus leaking them and causing the command processes in the pipeline to block on read / write.

Question

How would you prevent FDs from leaking to any child processes started in another thread in Raku?

Thanks

Whosoever

Couldn't you use something like OO::Monitors to wrap the filehandles to prevent other threads from using it at the same time? – Gaily

I think OO::Monitors prevents multiple threads from entering a method of an instance of that class at the same time. In my case, I have no control over what other parts of the system do; a program could be using some module that starts a thread that runs an external program via Proc::Async periodically, and I'll have no control over that. – Whosoever

Yep, that's why I said "like". It's more or less the same concept. OO::Monitors creates monitors that are able to do that; you would need to encapsulate filehandles in that kind of thing to avoid them "leaking". Someone might still access them via memory, but that would be hard to do. – Gaily

But that will only protect my module from leaking the FDs itself. I was wondering if there's a way to lock down the critical section of my module so that any attempts at forking the Raku process would be blocked until the FDs are closed. Hmm..., perhaps I can monkey patch run, shell, Proc and Proc::Async to make them acquire a lock that my module acquires for the critical section, assuming those are the only entrypoints in Raku that would create sub processes. – Whosoever

(My upvote is to signal it sounds like a good idea, not an affirmation that those four are the only routines/classes that need fiddling. Could a NativeCall lead to circumvention of such protection? Does that kibosh the idea?) – Autocrat

NativeCall can certainly circumvent the protection; however, given that using NativeCall to fork the Raku process is problematic to begin with, I wouldn't worry about it too much. I suspect monkey-patching Proc::Async might be enough. – Whosoever

A tangent. "Since there's no fork available in Raku, nor is it recommended due to threads in MoarVM". I'm not disagreeing, but are you aware that this works?: use nqp; my $i = nqp::fork(); say $i;. See A future for fork(2). – Autocrat

Hmm, interesting. I didn't know you can fork via nqp. I suppose it takes care of any problems one might have with calling the OS fork via nativecall? If so, then I might be able to use it to replace the hack of using Proc::Async + nativecall exec with just nqp::fork + nativecall exec. Thanks for bringing this to my attention! – Whosoever

"takes care of any problems one might have with calling the OS fork via nativecall?" No. I urge you to read the article and contact Bart Wiegmans. His approach relies on ensuring there's only one system thread in use at the time the fork occurs: "force MoarVM into a single active thread even with multiple user threads active". However "that still leaves cleanup to deal with, after returning from fork() in the child process. Also, this will not work if a thread is blocked on NativeCall." Also: "abusing the GC in order to enable a fork() may be a bit over the edge of insanity :-)". – Autocrat