Sending emails asynchronously: spool, queue, and cronjob/daemon

Question

I want to send emails asynchronously for faster and lighter http responses, but I'm struggling with many new concepts.

For example, the documentation talks about spool. It says I should use spool with a file, and then send emails with a command. But how should I be running that command? If I set a cronjob to execute that command every 1 minute (the minimum available in cron), users will have to wait an average of 30 secs for their emails to be sent (eg, the registration email).

So I thought of using a queue instead. I'm already using RabbitMQBundle for image processing (eg, thumbnail creation). But I only use this one periodically, so it is consumed from within a cronjob.

Maybe I should create a daemon that is always waiting for new messages to arrive the email queue and deliver them ASAP?

Fellow

What's the problem with 30sec delay? It's exactly as you said: a cron job execute a command every 1 minute, and the command itself is going to elaborate the queue. – Substituent

@Gremo The problem is that if there isn't much load in the server, I should be able to send the registration emails immediately. The same happens with image processing, imagine that I accept image uploads from the users. Making them wait 30 secs (let alone 1 min) for each submission will hurt the user experience. – Fellow

Then there is no need for a demon I think. You can spool and fire the command immediately and asynchronously from PHP itself. – Substituent

@Gremo That will create one process per email, not good. – Fellow

Answer 1

The solution is to send every email to a queue, and then consume that queue with a service. My service is very simple, it just takes items out of the queue, where each item is an array with from, to, body, etc., and sends that email. I'm using Thumper which makes Rabbit easier to use: github.com/videlalvaro/Thumper . And I make sure the service is always up using 'sv' (from Runit): smarden.org/runit/sv.8.html . You can use any other service or daemon manager you like.

Fellow

Answer 2

I have the same problem as you had. How you finally solved your problem?

For the moment I run a little script in the crontab in order to run in loop:

<?php
include('/var/www/vendor/symfony/symfony/src/Symfony/Component/Filesystem/LockHandler.php');
use Symfony\Component\Filesystem\LockHandler;

$lock = new LockHandler('mailer:loop');
if ($lock->lock()) {
    system('cd /var/www && php app/console swiftmailer:spool:send');
    sleep(1);
    $lock->release();
    shell_exec('cd /var/www && php LoopMailer.php > /dev/null 2>/dev/null &');
}

It's not very clean but it does his job.

Ocieock

Yes I solved it by using the Rabbit message queue I mentioned. Anything that is asynchronous I send it to a queue which is consumed either by a cronjob or a daemon (which I manage with the linux program "sv"). – Fellow

In my case I don't see any reason to use RabbitMQ. What is the purpose to use it, is the database not enough? I don't see also what RabbitMQ has to do with solving the 1 min problem? – Ocieock

With RabbitMQ I send the emails as soon as they are consumed in the queue, instead of every 1 min. – Fellow

But RabbitMQ is not able to send email, it's a simple queuing system. Therefor you must have a service consuming this queue?? I don't really get it... – Ocieock

Yes that's what "daemon" means. – Fellow

But what the advantage of RabbitMQ compare to the spool database? And could you give us more details concerning your service? Actually your service is the most interesting point of your solution ;-) – Ocieock

With spool and cronjob you have to wait an average of 30 seconds (between 0 and 60 seconds) for the mail to be delivered. With a queue and a daemon that consumes it they will be sent as soon as possible. My service is very simple, it just takes items out of the queue, where each item is an array with from, to, body, etc., and sends that email. I'm using Thumper which makes Rabbit easier to use: github.com/videlalvaro/Thumper . And I make sure the service is always up using 'sv' (from Runit): smarden.org/runit/sv.8.html . You can use any other service or daemon manager you like. – Fellow

Answer 3

You need 2 services one for spooling message and other for send instant emails. Check this

Ease

Thanks but this is not what I want. I want to spool/queue all the emails, to avoid injecting switfmailer (it's very heavy) on every request that triggers the sending of an email. I think the ideal solution is to add a message to the Rabbit message queue, and the daemon takes care of sending the email. This way, if I have say just 1 email request every 5 seconds, the queue will work very fast and will send emails instantly. If suddenly I get 999 email requests in 1 second, my server won't suffer: users will have to wait for their emails, but the http responses will still be blazing fast. – Fellow