Why avoid subshells?

Question

I've seen a lot of answers and comments on Stack Overflow that mention doing something to avoid a subshell. In some cases, a functional reason for this is given (most often, the potential need to read a variable outside the subshell that was assigned inside it), but in other cases, the avoidance seems to be viewed as an end in itself. For example

• union of two columns of a tsv file
  suggesting { ... ; } | ... rather than ( ... ) | ..., so there's a subshell either way.

• unhide hidden files in unix with sed and mv commands

• Linux bash script to copy files
  explicitly stating, "the goal is just to avoid a subshell"

Why is this? Is it for style/elegance/beauty? For performance (avoiding a fork)? For preventing likely bugs? Something else?

Gendron

This question is dangerously broad and opinion-based, but I think it's mainly for performance reasons. A subshell is forked in another process after all. – Cyton

@nwellnhof: I don't think it's broad or opinion-based to ask why a certain opinion exists. I think it would be broad if I asked why someone would hold this opinion (instead of why people do); and I think it would be opinion-based if I asked for people's views on subshells; but as it is, I would expect this to be pretty specific and answerable. – Gendron

One reason is performance. Forking a new shell is a non-trivial operation. – Contractor

Real subshells appear in the process list with the same name as the parent shell. For scripts using a lot of them (and let them run a long time) the process table is filled up with rather useless information. – Timberland

There are two reasons that I avoid subshells: performance and loss of environment variable data. If a subshell is invoked in any kind of loop, there is both a fork cost (to execute the subshell) and a setup cost (to go from running to ready to perform work) and a kill cost (to terminate it) for every invocation of the subshell. The bigger problem for me has always been that subshell variables die with the subshell which makes it very hard to get complex work results back from the subshell (often requiring temp files and adding yet more overhead). Also: PID changes with each subshell. – Flagstad

Answer 1

There are a few things going on.

First, forking a subshell might be unnoticible when it happens only once, but if you do it in a loop, it adds up to measurable performance impact. The performance impact is also greater on platforms such as Windows where forking is not as cheap as it is on modern Unixlikes.

Second, forking a subshell means you have more than one context, and information is lost in switching between them -- if you change your code to set a variable in a subshell, that variable is lost when the subshell exits. Thus, the more your code has subshells in it, the more careful you have to be when modifying it later to be sure that any state changes you make will actually persist.

See BashFAQ #24 for some examples of surprising behavior caused by subshells.

Gelation

Answer cited in Bash Scripting - How can I use a function to determine whether my variable will use file 1 or file 2?! Happy new year! – Argentina

Answer 2

sometimes examples are helpful.

f='fred';y=0;time for ((i=0;i<1000;i++));do if [[ -n "$( grep 're' <<< $f )" ]];then ((y++));fi;done;echo $y

real    0m3.878s
user    0m0.794s
sys 0m2.346s
1000

f='fred';y=0;time for ((i=0;i<1000;i++));do if [[ -z "${f/*re*/}" ]];then ((y++));fi;done;echo $y

real    0m0.041s
user    0m0.027s
sys 0m0.001s
1000

f='fred';y=0;time for ((i=0;i<1000;i++));do if grep -q 're' <<< $f ;then ((y++));fi;done >/dev/null;echo $y

real    0m2.709s
user    0m0.661s
sys 0m1.731s
1000

As you can see, in this case, the difference between using grep in a subshell and parameter expansion to do the same basic test is close to 100x in overall time.

Following the question further, and taking into account the comments below, which clearly fail to indicate what they are trying to indicate, I checked the following code: https://unix.stackexchange.com/questions/284268/what-is-the-overhead-of-using-subshells

time for((i=0;i<10000;i++)); do echo "$(echo hello)"; done >/dev/null 
real    0m12.375s
user    0m1.048s
sys 0m2.822s

time for((i=0;i<10000;i++)); do echo hello; done >/dev/null 
real    0m0.174s
user    0m0.165s
sys 0m0.004s

This is actually far far worse than I expected. Almost two orders of magnitude slower in fact in overall time, and almost THREE orders of magnitude slower in sys call time, which is absolutely incredible. https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html

Note that the point of demonstrating this is to show that if you are using a testing method that's quite easy to fall into the habit of using, subshell grep, or sed, or gawk (or a bash builtin, like echo), which is for me a bad habit I tend to fall into when hacking fast, it's worth realizing that this will have a significant performance hit, and it's probably worth the time avoiding those if bash builtins can handle the job natively.

By carefully reviewing a large programs use of subshells, and replacing them with other methods, when possible, I was able to cut about 10% of the overall execution time in a just completed set of optimizations (not the first, and not the last, time I have done this, it's already been optimized several times, so gaining another 10% is actually quite significant)

So it's worth being aware of.

Because I was curious, I wanted to confirm what 'time' is telling us here: https://en.wikipedia.org/wiki/Time_(Unix)

The total CPU time is the combination of the amount of time the CPU or CPUs spent performing some action for a program and the amount of time they spent performing system calls for the kernel on the program's behalf. When a program loops through an array, it is accumulating user CPU time. Conversely, when a program executes a system call such as exec or fork, it is accumulating system CPU time.

As you can see in particularly the echo loop test, the cost of the forks is very high in terms of system calls to the kernel, those forks really add up (700x!!! more time spent on sys calls).

I'm in an ongoing process of resolving some of these issues, so these questions are actually quite relevant to me, and the global community of users who like the program in question, that is, this is not an arcane academic point for me, it's realworld, with real impacts.

Cephalothorax

Wait, but the difference between your two examples is much greater than that one has a subshell. The one with the subshell is also calling an external program (namely grep), and it's also capturing the output of the subshell into a string. – Gendron

Yes, but it's something that would be, and is, very easy to do in order to achieve the same exact result in the test. In fact, I was doing just that, when I decided to test it and see what the actual speed/performance difference was. By carefully going through to find such traps, I was able to improve a very large script's performance by about 10%. Technically I could test a function in the script to see what that difference would be, though I did, and it's still quite substantial, as others noted, subshells are very expensive. I tend to fall into the habit for function string return in bash. – Cephalothorax

So, using your commands as a starting-point, I've tested four versions: (1) Bash regex, no subshell; (2) grep -q, no subshell; (3) Bash regex, superfluous subshell; and (4) grep inside a command substitution. I found that #2 took about three-fourths as long as #4, whereas #3 took only about one-fourth as long. So you're massively exaggerating the benefit of removing the subshell. – Gendron

It's always worth testing things. However, you're falling for a bit of pedantism in my opinion, which is everyone's right, the point I made was pretty clear and obvious, and the performance gain I saw was not a fantasy. I think sometimes it's possible to miss the forest for the trees, it's something it's worth being careful when dealing with these very binary non-natural systems. I like to question assumptions too. I'm unable to detect in your words anything that contradicts the facts I posted above, the data was pretty clear, between those two methods the difference in outcome was massive. – Cephalothorax

I updated with the grep -q, thanks, that's slightly better than grep in the subshell, about 70x slower. The first example was about 95x slower than the pure bash no subshell variant. So think before you use these testing methods, they are common, and easy bad habits to fall into. – Cephalothorax

Answer 3

well, here's my interpretation of why this is important: it's answer #2!

there's no little performance gain, even when it's about avoiding one subshell… Call me Mr Obvious, but the concept behind that thinking is the same that's behind avoiding useless use of <insert tool here> like cat|grep, sort|uniq or even cat|sort|uniq etc..

That concept is the Unix philosophy, which ESR summed up well by a reference to KISS: Keep It Simple, Stupid!

What I mean is that if you write a script, you never know how it may get used in the end, so every little byte or cycle you can spare is important, so if your script ends up eating billions of lines of input, then it will be by that many forks/bytes/… more optimized.

Cardcarrying

I'm surprised to hear your explanation of KISS as endorsing micro-optimizations ("every little byte or cycle"); in my experience, it's more often invoked to support the opposite view. – Gendron

well, my understanding of the word simple is etymological and means that's the complexity of the design shall be simple. Which most of the time is not synonym to easy. Here, using two fork()s when only one is necessary adds complexity. – Cardcarrying

KISS relates to simple design, not to not-computationally-complex algorithms - if you have to waste a couple more CPU cycles just to make your code easier to read and understand, KISS says you should do that. – Fatimafatimah

my interpretation of the KISS principle applied to Unix philosophy is that we shall always try to find a balance between readability and (non-)complexity. – Cardcarrying

@mgarciaisaia, indeed. However, shell in particular is full of pitfalls and side effects -- for instance, foo "$bar" is longer (more characters) than foo $bar, but more predictable, as it tells the shell to avoid string-splitting and glob-expansion processing phases. A more extreme example is while IFS='' read -r -d ''; do ...; done < <(find ... -print0), which looks like a substantial amount of overhead but vastly decreases room for bugs. Avoiding subshells is similar, inasmuch as it reduces the room for unexpected behaviors (see for instance mywiki.wooledge.org/BashFAQ/024). – Gelation

@CharlesDuffy: I'm not quite sure why you direct your comment at mgarciaisaia. Your points seem very much orthogonal to his. (foo $bar vs. foo "$bar" is not a matter of "design"; and anyway, I don't think anyone would argue that the former is easier to understand: it may look a bit simpler, but what it does is not exactly what it looks like it does.) – Gendron

@Gendron Sadly, a great many people make precisely that argument. I'm one of the regulars in freenode's #bash channel, and fighting that fight is almost a daily occurrence. – Gelation

Answer 4

I think the general idea is it makes sense to avoid creating an extra shell process unless otherwise required.

However, there are too many situations where either can be used and one makes more sense than the other to say one way is overall better than the other. It seems to me to be purely situational.

Silverts