Raku-native disk space usage

Question

Purpose:

• Save a program that writes data to disk from vain attempts of writing to a full filesystem;
• Save bandwidth (don't download if nowhere to store);
• Save user's and programmer's time and nerves (notify them of the problem instead of having them tearing out their hair with reading misleading error messages and "why the heck this software is not working!").

The question comes in 2 parts:

1. Reporting storage space statistics (available, used, total etc.), either of all filesystems or of the filesystem that path in question belongs to.
2. Reporting a filesystem error on running out of space.

Part 1

Share please NATIVE Raku alternative(s) (TIMTOWTDIBSCINABTE "Tim Toady Bicarbonate") to:

raku -e 'qqx{ df -P $*CWD }.print'

Here, raku -executes df (disk free) external program via shell quoting with interpolation qqx{}, feeding -Portable-format argument and $*CWD Current Working Directory, then .prints the df's output.

---

The snippet initially had been written as raku -e 'qqx{ df -hP $*CWD }.print' — with both -human-readable and -Portable — but it turned out that it is not a ubiquitously valid command. In OpenBSD 7.0, it exits with an error: df: -h and -i are incompatible with -P.
For adding human-readability, you may consider Number::Bytes::Human module

Garlic

Hi. Welcome to SO and the [raku] tag. :) This reads to me like an XY question. For example, on an OS that already has a df like program, the sensible thing would be to use it (but using run not qx for safety). On an OS without a df like program -- which is to say, a very immature OS -- Raku likely has no chance of compensating. But you haven't talked about such aspects. So perhaps your real purpose is to learn about Raku? Whatever it is, I think it would help a lot if you edited your question to add what's really behind your question. – Argonaut

@Argonaut Hi raiph. Thanks for your attention. Communication problems do exist, in this instance I frankly don't see any misstatements in my question. The bottom line "Purpose" may be somewhat heavy-worded but it describes exactly what I want. The snippet does exactly what I want, except not being native. OK, 1. My task at hand is: I need to download and save some stuff, then reshape it saving intermediate cache and finally save the result. This consumes storage space, so I find checking available space beforehand and during the course a worthy idea. [continued on the next comment] – Garlic

@Argonaut 2. I do understand run 'df', '-hP', $*CWD may be more safe since this form facilitates providing parts carefully, but is run «df -hP $*CWD» any more safe than qqx{}? You are free to throw some garbage at it as well! 3. I chose qqx{} over run because the former makes output available outright. I would split it and then check available space programmatically. [continued on the next comment] – Garlic

@Argonaut 4. Yes, I learn Raku, making glaring mistakes. 5. If the topic is beyond Raku, is it rather about OSes-and-filesystems architecture than about Raku? 6. Still, with Raku power, I shrug at the answers "no native". At least I hoped for a docs.raku.org/language/nativecall approach similar to docs.raku.org/language/… or for a modules.raku.org/search/?q=inline – Garlic

Hi again @uxer. I agree with you that there weren't any misstatements. More generally, there was nothing wrong with what you'd written. But the info you've added immeasurably improves your question. I've since encountered a GH issue (quite plausibly opened/closed by you, but it doesn't matter either way) that's an excellent motivating example. It's avoidable pain and annoyance; surely we can (and must) do better? – Argonaut

"I hoped for a ... nativecall approach ... or ... inline". Ahh. The English word "native" is decidedly ambiguous; "nativecall" is not. I wonder if there's just confusion over that? You could use NativeCall to call, say, statvfs() on a platform that supports that. Likewise you could use an Inline to call a foreign language module that does what you want. That said, why not use FileSystem::Capacity? – Argonaut

(Returning to this now closed issue in case it is the one that led you to explore what might be done better, I'd say the key issue there isn't reporting disk space but instead reporting a filesystem error. Reporting disk space would be nice, but failing to report running out of space, if that's what happened, is, on the face of it, terrible.) – Argonaut

"is run «df -hP $*CWD» any more safe than qqx ...? Potentially, if the current path contains unusual characters. qqx uses shell, which processes shell escapes, which an odd path name might include. As the doc for shell notes, "Shell escapes are a severe security concern ... Use run if you want to be safe." One still needs to be leery of injection attacks when using run, but at least issues due to using a shell are eliminated. See also github.com/Raku/doc/pull/3696. – Argonaut

"the programmer may need to make sure that such error/exception messages make their way to the user" Yes. Raku will throw exceptions, but it's up to the coder to decide how they wish to handle them. That said, it's pretty easy to get right. Fwiw, assuming your post began due to errors with Rakubrew, let me note Rakubrew is (necessarily) written in Perl, not Raku, and it looks like errors relevant to your case are getting dropped on the floor here, leading to the bogus "Done...". – Argonaut

Answer 1

Part 1 — Reporting storage space statistics

There's no built in function for reporting storage space statistics. Options include:

• Write Raku code (a few lines) that uses NativeCall to invoke a platform / filesystem specific system call (such as statvfs()) and uses the information returned by that call.

• Use a suitable Raku library. FileSystem::Capacity picks and runs an external program for you, and then makes its resulting data available in a portable form.

• Use run (or similar¹) to invoke a specific external program such as df.

• Use an Inline::* foreign language adaptor to enable invoking of a foreign PL's solution for reporting storage space statistics, and use the info it provides.²

Part 2 — Reporting running out of space

Raku seems to neatly report No space left on device:

> spurt '/tmp/failwrite', 'filesystem is full!'
Failed to write bytes to filehandle: No space left on device
  in block <unit> at <unknown file> line 1

> mkdir '/tmp/failmkdir'
Failed to create directory '/tmp/failmkdir' with mode '0o777': Failed to mkdir: No space left on device
  in block <unit> at <unknown file> line 1

(Programmers will need to avoid throwing away these exceptions.)

Footnotes

¹ run runs an external command without involving a shell. This guarantees that the risks attendant with involving a shell are eliminated. That said, Raku also supports use of a shell (because that can be convenient and appropriate in some scenarios). See the exchange of comments under the question (eg this one) for some brief discussion of this, and the shell doc for a summary of the risk:

All shell metacharacters are interpreted by the shell, including pipes, redirects, environment variable substitutions and so on. Shell escapes are a severe security concern and can cause confusion with unusual file names. Use run if you want to be safe.

² Foreign language adaptors for Raku (Raku modules in the Inline:: namespace) allow Raku code to use code written in other languages. These adaptors are not part of the Raku language standard, and most are barely experimental status, if that, but, conversely, the best are in great shape and allow Raku code to use foreign libraries as if they were written for Raku. (As of 2021 Inline::Perl5 is the most polished.)

Dogmatism

"Suggested non-native alternative approaches to df: use NativeCall to call, say, statvfs()". NativeCall and statvfs() is a native approach. NativeCall is built in. With it one can write Raku code that "magically" interfaces with native features available on the platform the program is running on. (You don't write C but do need to know enough C to get by.) statvfs is a native feature (system call) available on *nix OSes. If you can't get it to work yourself you could ask another SO question asking how to call statvfs() and use its data. – Argonaut

@Argonaut "I shrug at the answers "no native". At least I hoped for a … nativecall … or … inline" — "English word "native" is decidedly ambiguous; "nativecall" is not. I wonder if there's just confusion over that?" — my reading of your words in that comment is that NativeCall is not native. So, yes, there is quite a confusion over that. – Garlic

@Argonaut Isn't the definition of native somewhat blurred in Raku's NativeCall case? It is not Raku, it is C, so it is not native, but it is a built-in magic, so it is native? So how native exactly is NativeCall? Isn't it a special category of nativeness? The answer has been posted as a community wiki, so those knowledgable in the subject are welcome to amend the writing themselves. Thank you – Garlic

Hi @uxer. Thanks for your patience. When I wrote "NativeCall is not", I meant NativeCall is not ambiguous. Unfortunately English is, and it seems I accidentally confused you about what I suspect was earlier confusion on Liz's part (English is not her first language). – Argonaut

"how native exactly is NativeCall?" If "native" means C code or system calls, then very! It's entire point is to allow calling using C calling conventions, and then using a call's results in Raku. For example, if your system supports statvfs() or similar you could call that and use its info. (At least, afaik; I'm not a C hacker, and you'd need to know enough C to understand statvfs(), even though the code that calls and uses it would be written in Raku.) "Isn't it a special category of nativeness?" Yes. It allows calling and use of native code without needing to write native code. – Argonaut

Answer 2

raku -e 'run <<df -hP $*CWD>>'

If you're just outputting what df gives you on STDOUT, you don't need to do anything.

The << >> are double quoting words, so that the $*CWD will be interpolated.

Neill

My question is how to turn raku -e 'qx{ echo "from external program" }.print' into raku -e 'put "from Raku"', but with storage space – Garlic

If you're referring to the information of df being available natively to Raku without calling the df external program, the answer is: "no, Raku doesn't do that". – Neill