xargs command length limits

Question

I am using jsonlint to lint a bunch of files in a directory (recursively). I wrote the following command:

find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'

It works for most files but some files I get the following error:

Linting: ./LONG_FILE_NAME.json
fs.js:500
 return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                ^
  Error: ENOENT, no such file or directory '%'

It appears to fail for long filenames. Is there a way to fix this? Thanks.

Edit 1: Found the problem.

-I replstr

Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the con-structed arguments to utility, up to 255 bytes. The 255 byte limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x.

Edit 2: Partial solution. Supports longer file names than before but still not as long as I need.

find ./config/pages -name '*.json' -print0 | xargs -0I % sh -c 'file=%; echo Linting: $file; jsonlint -V ./config/schema.json -q $file;'

Hippogriff

Looks like the issue is with jsonlint, not xargs – Shakira

It is xargs, I figured out the problem just not a great solution. See edits. – Hippogriff

You can tell xargs to process one json file at a time by passing -n 1. Or maybe some number of files less than the breaking point at a time by passing -n 20 for example.. – Celisse

I don't think that will help. When using the -I flag with xargs, the limit for the command is 255 characters. So if the command is 78 characters long, the file name can only be 177 characters. – Hippogriff

If you removed the echo, you would be able to forego the -I. find ...|xargs jsonlint -V ./config.schema.json -q Another option is just to do this in a for loop. for file in `find ...`; do echo "Linting: $file"; jsonlint -V .... $file; done; – Shakira

Answer 1

On BSD like systems (e.g. Mac OS X)

If you happen to be on a mac or freebsd etc. your xargs implementation may support option -J which does not suffer from the argument size limits imposed on option -I.

Excert from manpage

-J replstr
If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not effect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir:
/bin/ls -1d [A-Z]* | xargs -J % cp -Rp % destdir

If you need to refer to the repstr multiple times (*points up* TL;DR -J only replaces first occurrence) you can use this pattern:

echo hi | xargs -J{} sh -c 'arg=$0; echo "$arg $arg"' "{}"
=> hi hi

POSIX compliant method

The posix compliant method of doing this would be to use some other tool, e.g. sed to construct the code you want to execute and then use xargs to just specify the utility. When no repl string is used in xargs the 255 byte limit does not apply. xargs POSIX spec

find . -type f -name '*.json' -print |
  sed "s_^_-c 'file=\\\"_g;s_\$_\\\"; echo \\\"Definitely over 255 byte script..$(printf "a%.0s" {1..255}): \\\$file\\\"; wc -l \\\"\\\$file\\\"'_g" |
  xargs -L1 sh

This of course largely defeats the purpose of xargs to begin with, but can still be used to leverage e.g. parallel execution using xargs -L1 -P10 sh which is quite widely supported, though not posix.

Spohr

Thanks, the $0 approach worked. It's the best solution. – Endue

Answer 2

Even if you use "-n" to limit the number of arguments passed, you may still run into "xargs: command line cannot be assembled, too long" if the arguments themselves are larger than a specific limit.

On macOS (and possibly other compatible systems), there is a limit on the size of arguments that can be passed in using the "-I" flag. From the manual page (man xargs):

     -S replsize
             Specify the amount of space (in bytes) that -I can use for
             replacements.  The default for replsize is 255.

Bumping this number up (e.g. adding -S1024) can resolve this problem.

Romine

Answer 3

Use -exec in find instead of piping to xargs.

find ./config/pages -name '*.json' -print0 -exec echo Linting: {} \; -exec jsonlint -V ./config/schema.json -q {} \;

Servomotor

Answer 4

The limit on xargs's command line length is imposed by the system (not an environment) variable ARG_MAX. You can check it like:

$ getconf ARG_MAX
2097152

Surprisingly, there doesn't not seem to be a way to change it, barring kernel modification.

But even more surprising that xargs by default gets capped to a much lower value, and you can increase with -s option. Still, ARG_MAX is not the value you can set after -s — acc. to man xargs you need to subtract size of environment, plus some "headroom", no idea why. To find out the actual number use the following command (alternatively, using an arbitrary big number for -s will result in a descriptive error):

$ xargs --show-limits 2>&1 | grep "limit on argument length (this system)"
POSIX upper limit on argument length (this system): 2092120

So you need to run … | xargs -s 2092120 …, e.g. with your command:

find ./config/pages -name '*.json' -print0 | xargs -s 2092120 -0I % sh -c 'echo Linting: %; jsonlint -V ./config/schema.json -q %;'

Spahi