Laravel queue with supervisord opening too many FIFO files

Question

I'm running a Laravel queue job called "webhooks" using beanstalkd and supervisord on my Ubuntu server. I can see the job properly running with process ID 4403:

webhooks                         RUNNING    pid 4403, uptime 4 days, 19:47:01

As you can see, this job has been running for 4 days. In my error logs, I started to notice the following error appearing:

error:02001018:system library:fopen:Too many open files

When I ran lsof | php to see what files were open, I noticed a ton of files open that had type FIFO. Here's an expert from the output:

COMMAND     PID   TID       USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME     
php        4403             root    0r     FIFO                0,8      0t0    9215811 pipe
php        4403             root    1w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    2w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    3w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    4w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    8r     FIFO                0,8      0t0    9215811 pipe
php        4403             root    9r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   10r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   11r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   12r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   13r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   14r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   15r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   16r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   17r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   18r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   19r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   20r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   21r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   22r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   23r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   24r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   25r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   26r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   27r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   28r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   29r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   30r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   31r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   32r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   33r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   34r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   35r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   36r     FIFO                0,8      0t0    9215811 pipe

This is just an excerpt. There are actually around 1200 of these FIFO files open.

Does anyone know what is causing the files to be created? What types of code would cause the system to open a new FIFO file? What are these files used for?

Workingwoman

A quick Google lets you know that FIFO files are named pipes. Something in your script, or in supervisord is creating these pipes. Have a look at named pipes in the PHP manual and see if you can't track down the code creating the pipes. – Hypopituitarism

Do you know what types of code can cause the system to create named pipes? I know that's a broad question, but I'm not sure where to start. The code that's running is syncing objects between my internal system, and an external system using the Bigcommerce API using their PHP SDK. So it's constantly pushing and pulling information from Bigcommerce. Previously I was opening too many connections, causing a similar too many files open error to occur, but that is now resolved and this FIFO problem has taken its place. – Workingwoman

Not familiar with either beanstalkd or supervisord, but I'd personally start by searching your code base for proc_open or mkfifo. – Reprisal

I don't call those functions anywhere myself. If they are being used it's because Laravel itself is using them. – Workingwoman

Post the code for webhooks – Psychomancy

Answer 1

I ended up increasing the file open limit on my system, and that seems to have fixed the problem. The supervisor docs say that:

supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS

After increasing my file open limit to 10,000, I have stopped seeing the error. I periodically check the number of open FIFO files, and there are times it's very high (in the thousands) and other times it's much lower (in the hundreds). So it looks like supervisor opens and closes these files constantly, and I just needed to make sure the system was allowing supervisor enough space to do its job.

In case anyone is wondering, I increased the file limit by adding the following two lines to the bottom of /etc/security/limits.conf.

root             soft    nofile          10000
root             hard    nofile          10000

I had to restart supervisor as well as log in and out of my system a few times before the new limit took effect, but eventually it worked.

Workingwoman