Is it possible to increase the maximum number of characters that ksh variable accepts?

Question

This is a follow up question to

What is the maximum number of characters that the ksh variable accepts?

I checked my environment and it's allowing only

#include <sys/limits.h>
$ cpp <<  HERE | tail -1
> #include <limits.h>
> ARG_MAX
> HERE
1048576

Is there a way to increase this? Or any alternatives for

 while read line;
   do
      #parse logic
   done < $filename

To handle really long lines? Based from the records I'm parsing it will not stop at 2M character lines.

Environment Details :

 AIX $ KSH Version M-11/16/88f 

Antipodal

Why exactly do you want so big shell variables? – Warbeck

And is it an AIX kernel, or a Linux one? – Warbeck

Can't you parse your file in something else than ksh? – Warbeck

Answer 1

You could compile a Linux 3.7.x kernel, and edit its include/uapi/linux/limits.h file to increase the ARG_MAX argument (to some bigger power of two, e.g. 2097152). But you should rather have a lot of RAM (e.g. 8GBytes) if you want to increase it more.

The actual limit is related to execve(2). That man page has a paragraph on it.

But you could probably avoid having huge shell variables (in the Unix environment). Did you consider using some other tool (awk, python, perl ....) to read your file? Their variable environment is not the shell environment transmitted to forked programs, so they can have variables with very long values. Maybe ksh has some builtin (unexport) to avoid exporting some variable into the Unix environment.

Warbeck

Hi Basile, thanks for your response. I don't have liberty to recompile the kernel. One question, does this apply to all scripting languages? If I use perl will I also encounter this when doing a read line? – Antipodal

I am parsing a large file and some lines have that many records. The script is working 99% of the time, but for 1% of the records it has 2M+ characters per line. – Antipodal

ARG_MAX doesn't come into play here, because it limits the length of the command line when invoking external utilities, whereas read is a shell builtin (and the data is read from a file, not passed as a command-line argument). The following ksh command demonstrates that it is possible to read lines that are longer than ARG_MAX: IFS= read line < <(printf "%$(( $(getconf ARG_MAX) + 1 ))s"); echo "${#line}" – Publisher