Symfony messenger workers do not stop when running under supervisor

Question

I am facing an odd behavior of the Symfony Messenger component. I set it up according to the documentation and I am issuing a messenger:stop-workers signal on each deploy as instructed here. However, a bug occurred in our system which I traced back to the fact that an old version of the code was being used by the Messenger workers.

Upon some more investigation, this is what happens in our setup:

• A worker is running, managed by supervisor.
• Just for debugging of this particular case, I start a new worker in the terminal app/console messenger:consume --env=prod -vv async to see what happens
• I issue the stop command app/console messenger:stop-workers --env=prod
• I would now expect that both the workers would be stopped (and supervisor would restart the one it's handling). However, that does not happen. The "debugging" worker does stop, but the one running under supervisor does nothing.

The supervisor-managed workers are limited to 1 hour of process time, after which they are stopped and restarted. I can see in the supervisord.log that this works well. Every hour there are log entries about the processes stopping and starting. But there is nothing whatsoever about them stopping from the messenger:stop-workers command.

I'm looking for ideas on why this would happen. I read the implementation of the workers and the shutdown signal is sent through a cache, but I did not find any problems in our configuration of it.

Assyria

supervisorctl restart all on deployment isn't an option? – Curia

That's an option and it's what I have done for now, to fix the immediate issue. But that is (or at least feels) less graceful, as it kills the worker processes instead of going the intended way of sending the stop signal. And if this turns out to be the best way to do the deploy, then the Symfony documentation is misleading and should be updated because it does not warn about this at all. – Assyria

Maybe this will help: Behind the scenes, this command sends a signal to each worker that it should exit. But the workers are smart: they don't exit immediately, they finish whatever message they're handling and then exit: a graceful exit. To send this signal, Symfony actually sets a flag in the cache system - and each worker checks this flag. If you have a multi-server setup, you'll need to make sure that your Symfony "app cache" is stored in something like Redis or Memcache instead of the fs so that everyone can read those keys. symfonycasts.com/screencast/messenger/deploy-restarting – Curia

Answer 1

I was running into a similar problem.

To force the stop of the rest of the consumers, as you have seen, the command uses a cache pool. In my case (and probably yours too), is the filesystem pool which is stored into /your_symfony_app/var/cache/{env}/pools

So if you are using Deployer or any other deployment system that replaces a symbolic link with every new deployment, you need to execute the command messenger:stop-workers inside the folder of your previous release.

Another option is to configure a cache pool shared by all the releases, like memcached or redis.

In my case, using Deployer (with a software still in development) I have been able to solve it by declaring a task like this, and putting it inside the deploy main task:

task('messenger:stop', function () {
    if (has('previous_release')) {
        run('{{bin/php}} {{previous_release}}/bin/console messenger:stop-workers');
    }
})->desc('Stop workers');

Monostich

Thanks for your answer! Where would you put messenger:stop in your Deployer flow? As the first thing? – Liliuokalani

In this case, I have it just right after the "deploy:symlink" task. – Monostich

I like this suggestion and it should work but unfortunately it didn't for me. The workers keep running, even though I see the stop command being executed in the previous release. Interestingly it DOES work if I manually issue the stop command in the previous release after deployment. I assume there are more finer details about the cache path and prefix seed here. In any case my "pkill" suggestion works for now. – Schedule

thanks @Monostich you just saved me pulling my hair out - thx! :D – Overthrow

@MathiasBrodala Unfortunately, messenger:stop-workers will gracefully stop them which means any long running tasks will have to finish first. There are ways around this but symfony doesn't expose any sort of API to listen on a 'stop-workers' event in your code to stop a long-running task in its tracks. You will have to get clever to get around this barrier. – Harijan

Answer 2

Here a workaround for this problem while we are waiting for better solutions.

You can use this workaround only if your Symfony Messenger supports the --failure-limit option committed here: https://github.com/symfony/symfony/pull/35453/commits/ea79206470ac3b71520a35129d36ca0d11ce4a09

1. Launch messenger through supervisor always with one max failure: php bin/console messenger:consume async --failure-limit=1

2. In your code base define a Message RestartMessenger and the corresponding Handler RestartMessengerHandler that simply throws an Exception such like MessengerNeedsToBeRestartedException

3. Create a Symfony command called app:messenger-restart-request that dispatches RestartMessenger

4. In your deploy script (bash, Ansible or others) add as last step: php bin/console app:messenger-restart-request. This will throw an exception that will cause the restart of the Messenger because --failure-limit=1

Sardella

Interesting idea! Though this will result in the messenger workers restarting every time an actual exception is thrown in the code due to a bug or whatever. – Assyria

there is special internal symfony excpetion for that "StopWorkerException" – Snapper

Answer 3

Another option is pkill in case it is available and allowed to be used:

desc('Stop Messenger workers');
task('messenger:stop-workers', function (): void {
    run('pkill --uid {{remote_user}} --echo --full messenger:consume');
});
after('deploy:symlink', 'messenger:stop-workers');

This reliably (and still cleanly) terminates all Messenger workers, independent from any release. The arguments in detail:

• --uid {{remote_user}} limits the matching to processes of the SSH user
• --echo outputs the name and former PID of the terminated processes
• --full also includes the command arguments for matching
• messenger:consume matches the .../console messenger:consume <flags> command line

This makes a few assumptions:

• one did ->set('remote_user', '...') on the host() to deploy
• the Symfony console messenger:consume processes are started with the same user by Supervisord
• there is only a single project on the server per user

Schedule

This is an interesting alternative, thanks! – Harijan

Answer 4

I added following task to Ansistrano after_cleanup_tasks.yaml file:

- name: Stop running Symfony Messenger consumers
  shell:
    chdir: "{{ ansistrano_release_path.stdout }}/../../current"
    cmd: php bin/console messenger:stop-workers

Fleabane