PHP, sendmail and transports - how to speed up mail sending

T

3

6

I just wrote a set of bulk-emailing classes for handling enormous amounts of emails and parsing their content according to passed params. If I test an email on 1000 random recipients and 1000 random senders from my database, up until the point the script hits the send() part (I commented it for now), I get performance of around 2 seconds, and 20 MB peak memory, which is great.

However, if I uncomment the send part, the sending takes 30 seconds. This is unacceptable, and I would like to speed it up somehow. It is obvious from testing that the delay is caused by none other than the $mail->send() call, as if it's waiting for it to return something before continuing the loop and sending the next email.

What I'm wondering is: how do I speed up the send() call? What can I do to make it faster? I tried using two sending methods:

Zend SMTP transport, connecting to the server directly and just sending. This takes 30 seconds per 1000 emails.
Sendmail via Zend_Mail. Simply calling Zend_Mail's send function after preparing each email. This takes 60 seconds.

Please note that queueing is definitely an option, and I have built it into my classes. All it takes is activating a cron and it works like a charm. But I'm wondering about the actual sending and how to speed that up. So, the actual send() call.

Talebearer answered 12/7, 2011 at 7:19 Comment(0)

S

2

I'd save the mails in a directory and send them using a shell script (cron/daemon/...):

Zend_Mail::setDefaultTransport(
    new Zend_Mail_Transport_File(
        array(
            'path' => __DIR__,
            'callback' => function() {
                do {
                    $file = 'email-' . date('Y-m-d_H-i-s') . '_' . mt_rand() . '.eml';
                } while (file_exists($file));
                return $file;
            },
        )
    )
);

Socha answered 12/7, 2011 at 13:10 Comment(9)

Correct me if I'm wrong, but a good MySQL query will always outdo a file reading. The queueing mechanism is already done either way, and needs no improvement right now. Likewise, our emails are each with a different content. So while we are sending 1000 emails in 2 seconds, each of those has a different recipient, different sender and different content. I believe this aspect you displayed here can be useful in bulk emailing only, yes? Where the content is identical in all the messages? If not, the question still remains - could this be faster than queueing in the database? I'm skeptical. – Talebearer 12/7, 2011 at 13:47

I thought that the problem is SENDING not GENERATING! :) There may be some optimised mail server, that would be able to bulk-load gazzilions of messages. Anycase - it would be faster than the php's sendmail method. Direct server access should be many times fater than php will ever be (it can for example use persistent connection). – Budd 12/7, 2011 at 16:48

Hmmm, quite right. A prepared email could easily be saved to disk in eml format, and from then on a script would take over. This could also serve decently as an alternate queue. Do you have any examples of this being done in an efficient manner? I would love to have a look at some. – Talebearer 13/7, 2011 at 6:27

No, actually not. I use the above code to dump mail instead of sending on development machine. I guess you can have a RAMDisk/SSD for saving those emails and then the only thing you need is to find a server capable of sending from directory at decent speeds ;) – Budd 14/7, 2011 at 9:54

But for extreme preformant solution I'd suggest bypassing Zend completly and dump the email yourselves, because there are some tests, encoding, etc. that you may not need which slow down the email creation. But doing that now, would be premature optimalization. :) – Budd 14/7, 2011 at 9:55

I'd first try switching sendmail transport with above solution to see if sendmail is the bottleneck in the send() call. If filesaving is MUCH faster I'd go with that outmail directory solution. Otherwise I guess that it's ZF, what's slowing the sending down... ;) – Budd 14/7, 2011 at 9:57

happy to hear that ! :) Might be awesome to make it available as OSS (github?) some time later and probably make a mail adapter for ZF for that (that I would volunteer to do). I guess that would be useable for many other users ;) – Budd 19/7, 2011 at 14:15

I'll definitely consider it if the company lets me open source it. – Talebearer 19/7, 2011 at 22:42

@Talebearer I'm in a similar situation right now trying to speed up the sending time of my emails. i send about 4 emails a second at 150kbs an email using sendmail. Do you have an example of your Go programs that sends emails? It would be greatly appreciated! – Ossification 15/11, 2011 at 4:35

G

2

You will need to speed up the MTA on the server. I recommend Postfix and that you really read up on each setting so you know how to fine-tune it. For a commercial solution I've heard PowerMTA is a good choice but I've never tried it myself.

There's only so much performance you can squeeze out of one machine, but a regular dedicated server should be able to deliver a rather impressive amount of mail once you've configured it correctly. The biggest performance bottleneck is usually the disk drives where the mail queue is stored, so consider using SAS (10k or even 15k RPM) or SSD drives.

Gauntry answered 12/7, 2011 at 7:23 Comment(2)

I will definitely look into this. Our email output will be increasing significantly very soon, and such a step up might be quite necessary. – Talebearer 12/7, 2011 at 8:24

We have begun planning this approach, but we will also include Tomas' answer into our solution. It has actually proven incredibly awesome, so far. – Talebearer 17/7, 2011 at 14:37

S

2

I'd save the mails in a directory and send them using a shell script (cron/daemon/...):

Zend_Mail::setDefaultTransport(
    new Zend_Mail_Transport_File(
        array(
            'path' => __DIR__,
            'callback' => function() {
                do {
                    $file = 'email-' . date('Y-m-d_H-i-s') . '_' . mt_rand() . '.eml';
                } while (file_exists($file));
                return $file;
            },
        )
    )
);