How can I learn more about why my Laravel Queued Job failed?

Question

The Situation

I'm using Laravel Queues to process large numbers of media files, an individual job is expected to take minutes (lets just say up to an hour).

I am using Supervisor to run my queue, and I am running 20 processes at a time. My supervisor config file looks like this:

[program:duplitron-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/duplitron/artisan queue:listen database --timeout=0 --memory=500 --tries=1
autostart=true
autorestart=true
user=duplitron
numprocs=20
redirect_stderr=true
stdout_logfile=/var/www/duplitron/storage/logs/duplitron-worker.log

In my duplitron-worker.log I noticed Failed: Illuminate\Queue\CallQueuedHandler@call occurs occasionally and I would like to better understand what exactly is failing. Nothing appears in my laravel.log file (which is where exceptions would normally appear).

The Question

Is there a handy way for me to learn more about what is causing my job to fail?

Molasses

Add loglevel under [supervisord] in the supervisord.conf. You can start with debug (check more here: supervisord.org/logging.html). Usually the reason why it fails, because the code you run doesn't end with correct "exit status". You can also see it here: #28938222 Last thing, you may need to add stderr_logfile in your configs. – Petes

Answer 1

In the newer Laravel versions there's an exception column in the failed_jobs table that has all the info you need. Thanks cdarken and Toskan for pointing this out!

==== OLD METHOD BELOW

Here's what I always do, but first - make sure you have a failed-jobs table! It's well documented, look it up :)

1. Run the php artisan queue:failed command to list all the failed jobs, and pick the one you're after. Write the ID down.

2. Then, make sure to stop your queue with supervisorctl stop all duplitron-worker:

3. Lastly, make sure your .env setting for APP_DEBUG = true.

4. Then run php artisan queue:retry {step_job_1_id}

5. Now manually runphp artisan queue:listen --timeout=XXX

If the error is structural (and most are), you should get the failure with debug stack in your log file.

Good luck with debugging :-)

Clustered

This is perfect -- thanks! Now I just need to figure out why it works when I retry it (also... the failed tasks slowly get reprocessed correctly despite the fact that I have --tries set to 1 which is confusing me, but that's a separate question!) – Molasses

I guess one followup question / if you have suggestions you could add to your answer -- are there ways I could get the failure stack in a log file when running in bulk. I suspect part of the issue might be related to constrained resources or possibly timeouts, which aren't happening when only a single thread is running. – Molasses

Well, like you said - if it runs well solo, it's not script related. You narrowed it down, that's the good news Your best bet then would be to check the system and php logs. Something has to be in there! It's most likely a time-out, or memory shortage :) – Clustered

--tries=1 will try to run the job one more time if it fails, I.e a totally of 2 times for each job before it gets sent to the failed table. – Rosina

@Molasses I'm sure you found the answer by now, but I had a similar issue: My notification was working fine until I queued it. After following the steps above, I checked storage/laravel.log and found that one of the parameters passed into my Notification class, "$custom_message" was coming up as "undefined property". Turns out I needed the statement: "protected $custom_message;" before the "public function __construct...." line. – Simard

I don't know if there's an addition in the later versions, but in 5.3 the failed jobs table has a column named exception where you can see what actually failed, no need for the tricks above – Ardyce

what is timeout? milliseconds? seconds? minutes? – Bunion

Argh! I should not have to dig through the database to find out why a job failed. Isn't there a way to just output the exception on the commandline when running queue:listen in the foreground? There's gotta be a better workflow. – Yangyangtze

Answer 2

as @cdarken pointed out, the exception can be found in your database table failed_jobs column name exception. Thanks @cdarken I wish his answer would be an answer and not a comment.

Bunion

Answer 3

• Run these command to create a table failed_jobs in db

php artisan queue:failed-table
php artisan migrate

• Run queue worker php artisan queue:work --tries=2

• Check the exception reason in your database table failed_jobs you've just created.

Tandy

Answer 4

1. php artisan queue:failed (view failed jobs)
2. php artisan queue:retry 6d7e18a4-ed84-4228-8c2a-477da82f4b5f (retry the jobs by id)

Spokeshave

Answer 5

If you're using database than goto failed_jobs table and look for the exception there.

Amarillo

Where can I find such a table? Can you elaborate a bit on the answer? – Dundalk

@SimasJoneliunas in your database you will find that table and if you goto config/queue.php you will find at the end of file name of failed job table – Amarillo