Can I get a faster output pipe than /dev/null?

Question

I am running a huge task [automated translation scripted with perl + database etc.] to run for about 2 weeks non-stop. While thinking how to speed it up I saw that the translator outputs everything (all translated sentences, all info on the way) to STDOUT all the time. This makes it work visibly slower when I get the output on the console.

I obviously piped the output to /dev/null, but then I thought "could there be something even faster?" It's so much output that it'd really make a difference.

And that's the question I'm asking You, because as far as I know there is nothing faster... (But I'm far from being a guru having used linux on a daily basis only last 3 years)

Dashed

Answer 1

Output to /dev/null is implemented in the kernel, which is pretty bloody fast. The output pipe isn't your problem now, it's the time it takes to build the strings that are getting sent to /dev/null. I would recommend you go through the program and comment out (or guard with if $be_verbose) all the lines that are useless print statements. I'm pretty sure that'll give you a noticeable speedup.

Embryectomy

yeah, but the worst part is in C++ and I don't have the code. Still it's cool to know /dev/null is the best ;) – Dashed

Ahh, okay. You did say it was written in perl. ;-) Yes, /dev/null is as fast as an output file descriptor can possibly be without kernel hacking, and even then it'd be hard to beat. – Embryectomy

I picked this answer for the additions in comment. I can't accept both answers. stackoverflow wouldn't let me;) – Dashed

/dev/null can't be beat for speed. What's faster than: return count; – Petitionary

@Embryectomy wouldn't all these "if $be_verbose" conditions slow down the calculations? I always had this query: which is the most efficient way to deactivate the debugging messages without removing or commenting them out one by one? – Rhythmist

@Rhythmist Depends. If they're runtime checks then yes. If they're compile time checks then no. But even at runtime checking a local variable is orders of magnitude faster than unconditionally executing a system call into the kernel just to effectively do nothing and return (assuming /dev/null redirect). – Sulphuric

@Rhythmist In practice I often compile for debug and release by just changing the definition of a print macro. If you want to support e.g. --verbose flag then you'll need to at least incur the overhead of checking the state of a configuration variable. That's going to be the most efficient way. Other than removing the prints at compile-time completely (like with macro definitions), you'll always have to check whether or not to do the prints. If you use macros then the prints just won't be emitted in the code. – Sulphuric

Answer 2

I'm able (via dd) to dump 20 gigabytes of data per second down /dev/null. This is not your bottleneck :-p

Pretty much the only way to make it faster is to not generate the data in the first place - remove the logging statements entirely. The cost of producing all the log messages likely exceeds the cost of throwing them away quite a bit.

Bumgardner

Thanx for the test. I didn't expect it to be a bottleneck, rather just wondering... it's 23:15 where I sit. Good time for such thoughts :P – Dashed

Answer 3

Unrelated to perl and standard output, but there is null_blk block device, which is even faster than /dev/null. Basically, it is bounded by syscall performance and with large blocks it can saturate memory bus.

Millwater