Pipe buffer size is 4k or 64k?

Question

I read in multiple places that the default buffer size for a pipe is 4kB (for instance, here), and my ulimit -a tends to confirm that statement:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15923
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8 // 8 * 512B = 4kB
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

But when I use a little program to test the buffer size (by writing into the pipe until the write() blocks), I see a limit of 64kB!

See this program:

#include <stdio.h>
#include <unistd.h>
#include <limits.h>

int main(void)
{
    int tube[2];
    char c = 'c';
    int i;

    fprintf(stdout, "Tube Creation\n");
    fprintf(stdout, "Theoretical max size: %d\n", PIPE_BUF);
    if( pipe(tube) != 0)
    {
        perror("pipe");
        _exit(1);
    }
    fprintf(stdout, "Writing in pipe\n");
    for(i=0;; i++)
    {
        fprintf(stdout, "%d bytes written\n", i+1);
        if( write(tube[1], &c, 1) != 1)
        {
            perror("Write");
            _exit(1);
        }
    }
    return 0;
}

And its output:

$ ./test_buf_pipe 
Tube Creation
Theoretical max size: 4096
Writing in pipe
1 bytes written
2 bytes written
3 bytes written
4 bytes written
[...]
65535 bytes written
[blocks here]

It strongly suggests that the pipe buffer size is actually 64k! What is happening here??

Birkle

This question is now two years old and the size in question is now programmable: https://mcmap.net/q/328448/-pipe-buffer-size-is-4k-or-64k – Segura

Answer 1

The other answers tell you that the pipe size is 64 KB. The reason why PIPE_BUF is 4KB is that PIPE_BUF is the largest size for which writes are guaranteed to be atomic. See http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html

Swifter

Does this imply that the buffers can go up arbitrarily? But the guaranteed atomic write is limiited to 4K? – Krupp

Answer 2

It's programmable now

---

As of Linux 2.6.35 you can use fcntl(2) with the F_SETPIPE_SZ operation to set the pipe buffer up to /proc/sys/fs/pipe-max-size. This is by default 1 MB; see proc(5).

Segura

Yes, but no. The pipe size is advertized by ulimit is still 4kB on Kernel 3.2 (the one I have here). My question was about PIPE_BUF, which is what janneb answered: PIPE_BUF is a constant and is the maximum size of a write which will be atomic when performed (i.e., thread-safe ;)). – Birkle

Does this mean by setting more then 4KB you can still guarantee that the write is atomic? – Jackhammer

Answer 3

In my experience, the single write test produced a total size of 65536, yet when I wrote 2700 at a time, I could only write 16 times, and then the next attempt stalls. I figure that the 'atomic' write needs to be within one 4K block, and that for each of my writes, it goes to the next full block to satisfy the request. So, in effect, the useable pipe size depends on the size of your writes.

Allhallows

Answer 4

Looks like the kernel use up to 16 buffers which adds up to 64k. See this link for an explanation of the ulimit output vs actual buffer size

Mayer

Answer 5

That's right. Since the 2.6.11 kernel, the pipe size in Linux is 64kB. Why ulimit reports it as 4Kb, I'm not sure, but that's wrong.

Kopje

Rah, same link as the one I pointed in my question. Didn't see the line stating 64k... Anyway why ulimit states 4K is a mysery to me... – Birkle

Only ksh and bash ulimit builtin report that pipe size, but it's a not a limit as set by setrlimit, and it's not the size of the pipe buffers,it's just PIPE_BUF, the maximum size for which a write() to a pipe is guaranteed to be atomic. – Totemism