Queued Laravel jobs all fire simultaneously, and don't show up in the jobs table

Question

I am using Laravel queues for commenting on Facebook posts. Whenever I receive data from a Facebook webhook, based on the received details I comment on the post. To handle 100 responses at once from Facebook webhooks, I am using Laravel queues, so that it can execute one by one. I used the step by step process as mentioned in Why Laravel Queues Are Awesome

public function webhooks(Request $request)
{
    $data = file_get_contents('php://input');
    Log::info("Request Cycle with Queues Begins");
    $job = (new webhookQueue($data)->delay(10);
    $this->dispatch($job);
    Log::info("Request Cycle with Queues Ends");
}

and this is my job class structure

class webhookQueue extends Job implements ShouldQueue
{    
    use InteractsWithQueue, SerializesModels;

    private $data;

    public function __construct($data)
    {
        $this->data = $data;
    }

    public function handle()
    {
       //handling the data here 
    }
}

I am hitting the webhooks() function continuously, and all the jobs are working simultaneously but not in the queue. None of the jobs are being stored in the jobs table. I have given a delay but it is also not working.

And this is my laravel.log:

[2017-02-08 14:18:42] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:44] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:47] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:47] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:47] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:47] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:48] local.INFO: Request Cycle with Queues Begins  
[2017-02-08 14:18:55] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:18:55] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:18:55] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:18:59] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:00] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:00] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:00] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:01] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:01] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:01] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:01] local.INFO: Request Cycle with Queues Ends  
[2017-02-08 14:19:01] local.INFO: Request Cycle with Queues Ends

Horsemanship

Database queue did not work for me because it sent to different channel – Anisaanise

Answer 1

for use queue you should some work :

in .env file you should change QUEUE_DRIVER or QUEUE_CONNECTION (newer versions) from sync to database, so open .env and do the follow

QUEUE_DRIVER=database

or

QUEUE_CONNECTION=database

after it you should create queue table in your database with artisan command :

php artisan queue:table
php artisan migrate

and for make sure that no config cached

php artisan config:clear

and finally you should run your queue with php artisan queue:listen or php artisan queue:work

Klockau

Yes, i have done this, after php artisan:listen command, while data is processing, nothing is coming in cmd prompt – Horsemanship

i have got the solution, i didnt change queue_drive to database, thank you for replying. – Horsemanship

@Mahdi - Hey, the command should be php artisan queue:listen not php artisan:listen – Alic

i fix typo issue ... sorry for that .. @Alic – Klockau

if your problem solve , mark this as solution of your question @Alic – Klockau

Interestingly, I had it set to database and had to change it back to sync to make this work. Thanks for the right hint! – Styles

queue:listen and queue:work are just processing one queue item but I have 20 queue items in the database and more getting stacked. @MahdiYouseftabar – Brand

@AnkitJindal i think you issue is inrevlevent to this issue ! you can ask it in an another question – Klockau

Just info, for the newer Laravel version (I don't know which version has this applied, but I am using v7.23.2), QUEUE_DRIVER has changed to QUEUE_CONNECTION in the .env file – Visigoth

I was not generating this queue table. also needed to run php artisan config:cache. thanks a lot – Agio

@ImranHossain you should run config:clear instead of config:cache! new users will have more issue when they ran config cache – Klockau

Answer 2

I had the same trouble, if you are using Laravel 5.7 or above, use this in .env file

QUEUE_CONNECTION=database

Next, clear config cache like this

php artisan config:clear

Physicality

I had the same when function can't delay() because artisan cache. So clear artisan cache to resolved. – Resplendence

This helped me. Many blog articles doesn't say that you must clear cache after some config changes. – Flavorsome

if you never run the config cache command it will never cache the config, which is what you should do in dev (never run config cache) – Salep

My queue was not running because of this. thanks a lot – Agio

Answer 3

In my case, i use custom queue name for group my jobs.

ProcessCourseInteractions::dispatch($courseProcessing)->onQueue('course_interactions');

That queue is not executed by:

php artisan queue:work

and

php artisan queue:listen

i need specify queue name (Valid for work and listen):

php artisan queue:work --queue=course_interactions

Falcone

It's working for me. Thanks! – Rehearing

Answer 4

Update for Laravel 5.7:

In .env, set QUEUE_CONNECTION=database so that dispatched jobs go to the database driver.

Then:

 # Creates a migration for the database table holding the jobs
php artisan queue:table

 # Executes the migration
php artisan migrate

 # Kicks off the process that executes jobs when they are dispatched
php artisan queue:work

Brinkley

This also applies for redis - found in newer Laravel / Horizon that QUEUE_DRIVER wasn't enough. 2 hours lost! Thank you :) – Plural

Answer 5

Make sure your app is not in maintenance mode... I had mine in maintenance, but allowing my local ip address... I couldn't figure out why it was not running. I had to finally go debugging the WorkCommand to find out...

./artisan up;

Ishmul

Answer 6

The accepted answer was a problem for me, but I also wound up on this question for 2 other similar problems which I solved, and maybe they will help other people that wind up here.

Other problem 1: job creation (constructor) works, but job handler does not fire - ever.

• the ever is key here because, typically, if none of them ever fire, then it could be because your job code is modified and your queue should be restarted.

Other problem 2: job creation (constructor) works, but job handler does not fire - sometimes.

• when sometimes is true for you, then it may be that your jobs that are not firing are because they are happening while in a transaction, like a DB::beginTransaction.

Assuming I want the job to fire even during a transaction, I can do this:

/**
 * Create a new job instance.
 *
 * @return void
 */
public function __construct($errorInfo)
{
    $this->errorInfo = $errorInfo;

    // Queues won't run the handlers during transactions
    // so, detect the level, and if it is not 0, rollback to force job handling
    if (\DB::transactionLevel() != 0) {
        \DB::rollBack();
    }
}

BUT DON'T DO THIS unless you want to fire your job and force rollback. My situation is unique in that my job sends emails on FATAL errors, so I want it to fire because I have an error breaking the process anyway (rollback going to happen and is unplanned due to uncaught error).

Here's a situation when you wouldn't want to do this:

• your job sends an email to a user when payment is successful
• your payment could be successful, could not be
• depending on successful payment, you rollback or commit

You should structure your dispatch to happen AFTER you rollback or commit. I did not have that luxury for my job because it happens when I cannot predict (a FATAL error). But if you have control over it, like knowing your payment is successful, dispatch after you have committed, or exited all levels of transactions!

I am not sure of the behavior of triggering a job while in the transaction, and then rolling back or committing. It could be worked around if it didn't work properly by adding a delay, but that seems unreliable (guessing at how long to wait) unless it was a significant delay.

Rosalbarosalee

Thank you. you have saved my half an hour – Fikes

Yes In my case everything was correct Except I was not restarting the queue worker after the changes. 1) "php artisan queue:work" or 2) php artisan queue:restart – Smallpox

Answer 7

Set .env

• QUEUE_CONNECTION=database

Add Migrations

• php artisan queue:table
• php artisan migrate

If you use

• dispatch(new YourJob($order))->onQueue('queue_name')

Finally Run This Command

• php artisan queue:work --queue=queue_name

Froemming

Answer 8

I am seeing that you already have Queue table.

Try running php artisan queue:listen --tries=3 or php artisan queue:work etc.

Queue work is for executing only one Job per command. So if there are 20 jobs in the table you might have to run queue work 20 times. That's why you can run queue:listen command. But it eats up a lot of CPU.

In the server, you might want to run your queue listen with max 3 tries in the background. SSH to your server in the Terminal / Command Prompt. Then CD to your project directory where the artisan file lives. Run this command:

nohup php artisan queue:listen --tries=3 > /dev/null 2>&1 &

In this case jobs will automatically be processed in the background. You just have to dispatch the job. And I would recommend using failed-jobs table. If you are using background queue listner.

Hope this helps.

Alic

thank you for replying, i have got the solution, i didnt change queue_drive to database – Horsemanship

Answer 9

For future readers of that questions , if queues were working before but no longer , just try to delete everything from the jobs table in the database , that worked for me .

Catkin

Answer 10

Don't make the same mistake as me,

I was running php artisan queue:work in the wrong directory.

Only wasted 30 minutes, could have been longer.

Horoscopy

Answer 11

Simply set QUEUE_CONNECTION=database in .env file

Quelpart

Answer 12

All thing are set-up and still not work then make sure added schedule on crontab -e * * * * * cd /var/www/html/<project_name> && php artisan schedule:run >> /dev/null 2>&1

Mayhap

Answer 13

You can try again in the following way (I assume that you did instructions in Laravel docs but someday it's not working):

Step 1: drop table 'jobs' in your database.

Step 2: run command 'php artisan migrate' in console to create table 'jobs' again.

Step 3: run command 'php artisan queue:work' in console

Step 4: retry your app

Note that in .env file, you set up:

QUEUE_CONNECTION=database

QUEUE_DRIVER=database

P/s: It works for me!

Germanize