Why are my Laravel Queue Jobs failing after 60 seconds?

Question

The Situation

I'm using Laravel Queues to process large numbers of media files, an individual job is expected to take minutes (lets just say up to an hour).

I am using Supervisor to run my queue, and I am running 20 processes at a time. My supervisor config file looks like this:

[program:duplitron-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/duplitron/artisan queue:listen database --timeout=0 --memory=500 --tries=1
autostart=true
autorestart=true
user=duplitron
numprocs=20
redirect_stderr=true
stdout_logfile=/var/www/duplitron/storage/logs/duplitron-worker.log

There are a few oddities that I don't know how to explain or correct:

1. My jobs fairly consistently fail after running for 60 to 65 seconds.
2. After being marked as failed the job continues to run even after being marked as failed. Eventually they do end up resolving successfully.
3. When I run the failed task in isolation to find the cause of the issue it succeeds just fine.

I strongly believe this is a timeout issue; however, I was under the impression that --timeout=0 would result in an unlimited timeout.

The Question

How can I prevent this temporary "failure" job state? Are there other places where a queue timeout might be invoked that I'm not aware of?

Wizard

Check the max_execution_time your php.ini How much does it says? If it's 60 secs, there's your problem. try increasing the timeout. – Bobette

Great thought @WillyPt -- wouldn't the php.ini settings terminate the entire script, though? It continues to resolve. (FWIW max_execution_time was set to 30s, I'll explore and experiment along those lines). – Wizard

Answer 1

It turns out that in addition to timeout there is an expire setting defined in config/queue.php

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'expire' => 60,
    ],

Changing that to a higher value did the trick.

---

UPDATE: This parameter is now called retry_after

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 60,
    ],

Wizard

Thank you sir, this helps me solve my problem. But as pointed out by David's answer, "expire" now called "retry_after" – Restraint

I have implemented the queue for importing huge CSV, I will not go to the server every time to check the progress PHP artisan queue:work how to handle this case? @slifty? – Tsaritsyn

Answer 2

Important note: "expire" is now called "retry_after" (Laravel 5.4)

Bryson

Answer 3

This will work

php artisan queue:listen --timeout=1200

Adjust the time based on your need

Permian

I have implemented the queue for importing huge CSV, I will not go to the server every time to check the progress PHP artisan queue:work how to handle this case? @Munna Khan – Tsaritsyn

You can set a scheduler to keep the queue worker alive. Another way to configure supervisor on your server. – Permian

Yes, perfect thanks for the help! Supervisor should keep queue alive running. – Tsaritsyn

NOTE: If this is null (aka not set) there will not be a timeout for the QUEUE WORKER. The JOB timeout defaults to 60 seconds. That can be changed in the job which is not what the answer does. This answer looks to be conflating the two timeouts. – Chordophone

Answer 4

ok, this is an older question, but I've been recently having the same problem with Laravel 8x with jobs in the queue ending after 60 seconds. Changing the php.ini, adjusting queue drivers, changing supervisor and startup.sh settings didn't help.

What did help after some experimentation was adding:

public $timeout = 600;

after use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;inside the job itself. eg:

class QueueJob implements ShouldQueue
{
  use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

  protected $param_one;
  protected $param_two;

  public $timeout = 600; // <--------------------------- HERE

  /**
    * Create a new job instance.
    *
    * @return void
    */
  public function __construct($param_one, $param_two)
  {
    $this->param_one= $param_one;
    $this->param_two= $param_two;
  }

  /**
    * Execute the job.
    *
    * @return void
    */
  public function handle()
  {
     //Code goes here
  }

All of a sudden the queue no longer times out and is working as intended :)

(This is an expansion of @jonlink's answer from their comment above)

Hasheem

Answer 5

In my case, I am using Symfony\Component\Process\Process. I have to set timeout as following as well.

$process = new Process([...]);
$process->setTimeout(null);

Krumm