Running multiple commands with xargs

Question

cat a.txt | xargs -I % echo %

In the example above, xargs takes echo % as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:

cat a.txt | xargs -I % {command1; command2; ... }

But xargs doesn't accept this form. One solution I know is that I can define a function to wrap the commands, but I want to avoid that because it is complex. Is there a better solution?

Erbes

Most of these answers are security vulnerabilities. See here for a potentially good answer. – Sundae

I use xargs for almost everything, but I hate putting commands inside strings and explicitly creating subshells. I'm on the verge of learning how to pipe into a while loop that can contain multiple commands. – Eulogistic

Test the solutions on inputs like: ", *, a two spaces b, $(echo Do not print this). If these do not work as expected, there are likely other bugs in the solution, too. – Flavoring

Answer 1

cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

...or, without a Useless Use Of cat:

<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

---

To explain some of the finer points:

• The use of "$arg" instead of % (and the absence of -I in the xargs command line) is for security reasons: Passing data on sh's command-line argument list instead of substituting it into code prevents content that data might contain (such as $(rm -rf ~), to take a particularly malicious example) from being executed as code.

• Similarly, the use of -d $'\n' is a GNU extension which causes xargs to treat each line of the input file as a separate data item. Either this or -0 (which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quite shell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can use tr '\n' '\0' <a.txt | xargs -0 ... to get line-oriented reading without -d).

• The _ is a placeholder for $0, such that other data values added by xargs become $1 and onward, which happens to be the default set of values a for loop iterates over.

Bechler

For those unfamiliar with sh -c -- note that the semicolon after each command is not optional, even if it is the last command in the list. – Generous

At least on my configuration, there must be a space immediately after the initial "{". No space is required before the ending curly brace, but as Mr. Sussman noted, you do need a closing semicolon. – Alidia

This answer previously had curly braces around command1 and command2; I later realized they're not necessary. – Bechler

To clarify the above comments about semicolons, a semicolon is required before a closing }: sh -c '{ command1; command2; }' -- but it's not required at the end of a command sequence that doesn't use braces: sh -c 'command1; command2'` – Bechler

@A-B-B: Using && rather than ; might well be appropriate, but reducing the amount of escaping you have to do is not a good reason to do it, since it has different semantics. You'll probably enclose the command in single quotes anyway, so it doesn't make much difference. – Bechler

cat a.txt | xargs -I % sh -c 'command1; command2; ...' is great, except if the input contains shell comment character # (as it is likely to, if a file listing), in which case only part of the first command executes. – Edna

@KrazyGlew: Why would a file listing be likely to contain # characters? (Certainly it could.) – Bechler

@KeithThompson: emacs creates temporary files whose names contain #s. See emacswiki.org/emacs/AutoSave ... I could swear that many other tools used to do similar, but googling finds nothing. ... Although in modern usage, in GUIs, like MacOS, the user finds it very easy to create files containing special characters like blank, hash (#), and many other forms of shell punctuation. Not being able to handle such filenames is often a security flaw. – Edna

whatif the command1 or command2 itself contains ' or " quotes? – Timbal

@prehawk: Two possibilities. (1) This is your golden opportunity to show off your quoting skills. (2) Wrap the commands in a shell script. – Bechler

@KeithThompson I'm learning xargs because I need to find out some problem by filtering multiple files in multiple machines. I wrote perl commands to do this, and command is always changing thus I can't write it into script. Writing a script means I need to upload these changing command everytime. – Timbal

Could someone give this a use case? I've been trying ls /Users/fishema/Desktop/Projects/Bcore_clients/579ML/results/051116_results/JF_intersect_gene*|xargs -I % sh -c 'head -n2; tail -n1;' to no avail... – Mantelletta

sigh Doesn't work in fish. I love the shell, but some things in bash do make sense – Sally

If you're including the % character somewhere in your string passed to sh -c, then this is prone to security vulnerabilities: A filename containing $(rm -rf ~)'$(rm -rf ~)' (and that's a perfectly legal substring to have within a filename on common UNIX filesystems!) will cause someone a very bad day. – Serranid

Useless use of cat! Okay! I've seen the redirection way before but I never really internalized what was going on. I assume it's always okay to use that pattern? left angle bracket, filename, command....? – Lavender

@ToddWalton: Yes, generally the redirection can appear anywhere in the command. For example, < /etc/motd cat -n, cat < /etc/motd -n, and cat -n < /etc/motd are all equivalent. (At least for bash, but it appears to be the case for most shells.) – Bechler

@KeithThompson, would you be willing to accept an edit showing a practice that doesn't involve command injection vulnerabilities? – Serranid

@CharlesDuffy: I believe an edit would be applied whether I accept it or not. But yes, especially if it's reasonably clear. – Bechler

@KeithThompson, sure, it'd be applied, but if you disagreed we could be in an edit war, and that's not a good place for anyone. Anyhow, let me know if you think the current revision is clear enough or needs further work. – Serranid

I'll take a closer look at this in the next day or two. – Bechler

Shouldn't the for x do ... be for x ; do ...? – Lautrec

@Lautrec The semicolon is optional. That's not clear from the POSIX syntax, but the bash manual shows it as for NAME [ [in [WORDS ...] ] ; ] do COMMANDS; done. Apparently you don't need the semicolon if there's no in part. I'd probably use a semicolon myself, but I'm hesitant to change the answer. – Bechler

If I only have one line in a.txt, it won't do anything, besides, the command in the answer seems always ignore the first line? – Lyle

@Lyle Let me get back to you on this. It's been a while since I've looked at this answer. – Bechler

I'll speculate that @Lyle didn't include the (rather critical) trailing underscore: "The _ is a placeholder for $0, such that other data values added by xargs become $1 and onward...". I didn't notice it at first either. – Ritual

Answer 2

You can use

cat file.txt | xargs -i  sh -c 'command {} | command2 {} && command3 {}'

{} = variable for each line on the text file

Marxmarxian

This is insecure. What if your file.txt contains a datum with $(rm -rf ~) as a substring? – Serranid

This worked well for me, luckily none of the zoneinfo timezone definitions contain rm -rf ;) – Latham

+1. It's incredible how much effort people will spend on security where it is not needed (e.g. processing a list of IP addresses, PIDs or USB device names) – Acicular

And yet, as a "generic solution", the security concern should be (and was) rightly noted. The command should not be used on untrusted or unsanitized input unless you understand the risks. If you do trust your input, then have at it. – Nonstop

hm I get permission denied for commands running inside sh -c (or bash -c) – Prakash

@CharlesDuffy The security issue can be easily solved by surrounding the input with single quotes: cat file.txt | xargs -i sh -c "command '{}' | command2 '{}' && command3 '{}'". For example echo '$(echo foo)' | xargs -i sh -c "echo '{}'" returns the string $(echo foo) and not only foo, which would happen if it would have been get executed. Note: xargs does not accept single or double quotes as input (if someone tries to break this). – Nowak

@mgutt, not so simple as all because the input can also contain single quotes that cancel our the ones you're explicitly adding. And yes xargs does allow quotes in input; they just need to be escaped. – Serranid

@mgutt, here, you can try it yourself. printf '%s\n' $'hello\\\'$(touch evil)\\\'.txt' | xargs -I{} sh -c "echo '{}'" && ls -l evil – Serranid

@mgutt, ...also, the quote removal done by xargs (which, as discussed above, only happens when there's no escaping) is mode-dependent -- add on -0 in either BSD or GNU or -d $'\n' in GNU xargs and it goes away; it's not something that was ever designed as a security measure in the first place, but rather is intended to let someone write "first item" "second item" to pass two items on a command line as shell parsing would do. Thus, it should be completely unsurprising that, also like shell parsing, "first 'has quotes' item" only removes the outer and not inner quotes. – Serranid

Answer 3

With GNU Parallel you can do:

cat a.txt | parallel 'command1 {}; command2 {}; ...; '

For security reasons it is recommended you use your package manager to install. But if you cannot do that then you can use this 10 seconds installation.

The 10 seconds installation will try to do a full installation; if that fails, a personal installation; if that fails, a minimal installation.

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh

Flavoring

Installing tools via running random scripts from unknown sites is horrible practice. Parallel has oficiall packages for popular distros, which can be trusted (to some extend) way more than random wget|sh... – Susan

Let us see what is the easiest attack vector: Pi.dk is controlled by the author of GNU Parallel, so to attack that you would have to break into the server or take over DNS. To take over the official package of a distribution, you can often just volunteer to maintain the package. So while you might be right in general, it seems in this particular case your comment is not justified. – Flavoring

In general it is not more dangerous to run a shell script from an untrusted site than to download a tar file and install software from same site (at least if you do not check the signature). – Flavoring

In practice I do not know that pi.dk belongs to the author. Actually verifying that this is the case, thinking of how to use ssl in wget and checking that this command does what it is supposed to do is a bit of work. Your point that the official package can contain malicious code is true, but that also holds for the wget package. – Soule

This might not be the best solution if each of the commands OP wants to execute must be sequential, correct? – Teachin

@IcarianComplex Adding -j1 will fix that. – Flavoring

There's another thing to consider with your idea of linking to a site that isn't say 'standard'. Let's say you provide evidence it's the owner for this case. It has the potential to convince people that they can and even should use software from just any website. As @Soule points out the risk holds for both websites. Now this does not mean that it's never justified but still if there's a standard repository that people are familiar with it's a better idea. As also pointed out it is easier to verify. That is very important. That's entirely the point! – Selfexplanatory

The author of the package controls the domain until the domain fails to renew, is seized by a hostile actor who makes use of a ready-made attack vector. Or until someone breaches his site and adds hostile code. Or until someone realizes that despite generating 3 different checksums, none of the operations - as written - prevent running install.sh even if the sums do NOT match. Even if the sums do not match, a reader might conclude that since the post was made 10+ years ago, the shell script has been changed and has a new checksum NOT published here. – Alfonsoalfonzo

Answer 4

I prefer style which allows dry run mode (without | sh) :

cat a.txt | xargs -I % echo "command1; command2; ... " | sh

Works with pipes too:

cat a.txt | xargs -I % echo "echo % | cat " | sh

Eulogist

This works, until you want to use GNU xargs' -P option... (If not, I mostly use -exec on find, since my inputs are mostly filenames) – Cockerel

Fails on input: " – Flavoring

Answer 5

This is just another approach without xargs nor cat:

while read stuff; do
  command1 "$stuff"
  command2 "$stuff"
  ...
done < a.txt

Barnacle

Buggy, as given. Unless you clear IFS, it'll ignore leading and trailing whitespace in the filenames; unless you add -r, filenames with literal backslashes will have those characters ignored. – Serranid

Does not answer the question. It specifically asked about xargs. (This is hard to expand to do something similar to GNU xargs' -P<n> option) – Cockerel

This works perfectly well. You can also use it as a piped command like $ command | while read line; do c1 $line; c2 $line; done – Fragment

Answer 6

This seems to be the safest version.

tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

(-0 can be removed and the tr replaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargs with find with -print0 output) (This might also be relevant on xargs versions without the -0 extension)

It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$@"][1]

If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -i or -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)

If the commands can only take one parameter at a time:

tr '[\n]' '[\0]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

Or with somewhat less processes:

tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$@"; do command1 "$f"; command2 "$f"; done;' ''

If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:

tr '[\n]' '[\0]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

This should be robust against any special characters in the input. (If the input is null separated.) The tr version will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.

The blank first parameter for bash -c is due to this: (From the bash man page) (Thanks @clacke)

-c   If the -c option is present, then  commands  are  read  from  the  first  non-option  argument  com‐
     mand_string.   If there are arguments after the command_string, the first argument is assigned to $0
     and any remaining arguments are assigned to the positional parameters.  The assignment  to  $0  sets
     the name of the shell, which is used in warning and error messages.

Cockerel

This should work even with double quotes in filenames. That requires a shell that properly support "$@" – Cockerel

You are missing the argv[0] argument to bash. bash -c 'command1 "$@"; command2 "$@";' arbitrarytextgoeshere – Vickey

This is not about what xargs does. bash with -c takes first (after the commands) one argument that will be the name of the process, then it takes the positional arguments. Try bash -c 'echo "$@" ' 1 2 3 4 and see what comes out. – Vickey

It's nice to have a safe version that doesn't get Bobby-Tabled. – Sundae

Answer 7

One thing I do is to add to .bashrc/.profile this function:

function each() {
    while read line; do
        for f in "$@"; do
            $f $line
        done
    done
}

then you can do things like

... | each command1 command2 "command3 has spaces"

which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.

Stolid

Underrated answer, this is extremely handy – Jansen

Does not work correctly if input has two spaces in a row or *. – Flavoring

Answer 8

Another possible solution that works for me is something like -

cat a.txt | xargs bash -c 'command1 $@; command2 $@' bash

Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.

Example:

cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $@; echo "data again: " $@' bash

Maxa

If you don't quote "$@", then you're string-splitting and glob-expanding the argument list. – Serranid

Answer 9

My current BKM for this is

... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'

It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)

@KeithThompson's suggestion of

 ... | xargs -I % sh -c 'command1; command2; ...'

is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.

Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.

Example of the problem:

$ bash 1366 $>  /bin/ls | cat
#Makefile#
#README#
Makefile
README

Oops, here is the problem:

$ bash 1367 $>  ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README

Ahh, that's better:

$ bash 1368 $>  ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>  

Edna

# problem can be easly solved using quotes: ls | xargs -I % sh -c 'echo 1 "%"; echo 2 "%"' – Dinsmore

Answer 10

Try this:

git config --global alias.all '!f() { find . -d -name ".git" | sed s/\\/\.git//g | xargs -P10 -I{} git --git-dir={}/.git --work-tree={} $1; }; f'

It runs ten threads in parallel and does what ever git command you want to all repos in the folder structure. No matter if the repo is one or n levels deep.

E.g: git all pull

Laoag

Your example is very useful, but it's convoluted enough that an explanation would help. It appears that it doesn't answer the question about how to execute multiple commands with xargs. What your example does is git --git-dir=A1/.git --work-tree=A1 pull in your example where A1 is one of the repos it finds. The question was how to do something like ls -al {}; rm -f {} (i.e. two commands for each line given to xargs not one). – Alfonsoalfonzo

My comment seems to be way out of place. It does not answer the original question at all. I was certain I posted this on a different question :D What my comment does is to prepare a git alias. The alias allows for running git commands on all repos located in sub-directories. @SteventheEasilyAmused, do you think I should remove the comment? – Laoag

Answer 11

I have good idea to solve the problem. Only write a comman mcmd, then you can do

find . -type f | xargs -i mcmd echo {} @@ cat {} @pipe sed -n '1,3p'

The mcmd content as follows:

echo $* | sed -e 's/@@/\n/g' -e 's/@pipe/|/g' | csh

Forgat

I applaud you for not settling for the sh -c solution which doesn't sit right with me. – Eulogistic

Uhhhh, it's not like csh is less awful than ... anything at all. – Mercury