What methods exist for setting up a large email notification system?

Question

My company has a website built with PHP. We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.

This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.

Now I've looked at mass mailing services like MailChimp (which would replace our current system of sending the same email to many people), but what I think I'd really like to do is to set up a somewhat-sophisticated notification system.

Rather than send a mass email to each person each time something important happens, I'd like clients to be able to customize the rate and content of the emails that they receive.

Even using this new idea, we're talking about A LOT of emails being sent.

So my question is very specific: I have a rough idea of how to build the system internally, but what is the best way to send out all of these emails?

Bullet points to consider:

• Sometimes emails' contents are identical across recipients, but many of them will be customized per-user (they choose what they get notified about, and sometimes it is aggregated).
• I want a system that is not going to choke the server, and will complete in a decent amount of time. I don't mind going with a third-party service (even a paid one) if that is what it is going to take.
• The system should hook into PHP easily, or the API or whatever should be relatively easy for me to call from your typical web server.
• We have a dedicated server and full control over it (so we can install apps, services, whatever).
• Any kind of detailed tracking information (opens, clicks, etc) is a huge plus.
• These emails are sometimes time-sensitive (so can't take all day to send).

Thoughts? Tips? Point me in the right direction?

---

EDIT

To clarify:

I can do these on my own:

• maintain user list
• handle email content generation based on user preferences

And need something else (app, third-party service, w/e) to:

• accept email content and addresses and actually send the emails out
• provide tracking data (opens, clicks, etc). The more detail the better.

I'm leaning towards a third-party service, since I'm not sure any app can avoid choking the server when sending thousands of emails (though I wouldn't consider myself an email expert so I could be wrong).

Drawl

Answer 1

I'd recommend using the third party mailing service Silverpop, or something like it. We've used them for a few years and have been fairly satisfied. They already have relationships with the major email clients (AOL, Yahoo!, Gmail, etc.) and they do a good job of telling you if the stuff you're sending is likely to be classified as SPAM.

They have a fairly extensive API that uses XML HTTP/HTTPS requests that can tie in to existing systems. You can use it to remotely trigger emails, schedule mailings, customize email contents, set up, manage and query huge lists of recipients, run batch processes, etc.

It isn't a perfect service, but compared to a lot of others out there, they do pretty well. I have had very few complaints about them thus far.

Secundines

Answer 2

We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.

This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.

Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on.

When you say it chokes your server - do you mean your email server? Your web server? something else?

There's nowhere near enough information here to make a proper diagnosis but it looks like the problems are of your own making - sure, there are lots of people out there who promise to sort all your problems for you if only you buy their latest product/service. But there's a very good chance that this isn't going to solve your current problems.

Can you tell us:

1. what OS the PHP is running on

2. how you invoke the code to create the emails

3. what the mail config in the php.ini file is

4. what type of MTA are you using? On what OS?

5. how is youe MTA copnfigured - does it deliver directly or via a smart relay?

6. which server is getting "choked"?

7. What anti-spam measures do you have in place for outgoing mail?

Then tell us what you've done to diagnose the fault and why you think its specifically on sending mails.

C.

Tzar

Symcbean, thanks for being thorough. Always appreciated! I don't know all the configuration details off the top of my head, nor the intricacies of the code (I'm newly-employed). I'll delve a little deeper and look for some answers to your questions. I know we're running CentOS, and that there is only one machine running everything. By 'choked' I mean that our web server slows down and it takes upwards of 5 or 6 hours to send all the emails. I assumed this was a combination of bad coding and the fact that you can't send that many emails from a typical configuration. – Drawl

"Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on." And to clarify: my implication was never that the simple mailing function was the culprit; simply that either the code we were running or the server configuration itself was a bottleneck for the rate at which we could send emails. I'm not very familiar with email servers or configuration so I don't know what a typical web server can handle for outbound emails. – Drawl

We are using 'sendmail' for the emails, and it's the default settings ('sendmail -t -i'). – Drawl

You might want to try disabling dequeuing of messages in the MTA (i.e. actually sending them out of door) and see how long it takes to populate the mail queue, then switch on the dequeuing and see how long it takes to clear the backlog - that way you can seeprate problems with the MTA from elsewhere. If it is a problem on the MTA, then I'd recommend you start another topic on that. OTOH, if the MTA is still running slow, try commenting out the call to mail() in your PHP code and see how long it takes to run the script. HTH – Tzar

Answer 3

I usually got around this by having a mail "sending" function that dumped the emails into a queue (database table) with a job that ran every couple of minutes, grabbed the next x emails in the queue, sent those out and marked them as succeeded. That's the simple bones of it. You can then add on handling for email failures, returned mail, etc in version 2.

Misjudge

I'm currently using this method. My hosting company allows me to send up to 200 e-mails per hour, per domain. So with a cronjob I handle the queue and the timings. It works nice and it's somewhat simple to implement. – Oberheim

After posting this, the Dr. Jekyll side of my head made me come back and attach a caveat: before building this internal system, make sure to ask yourself why you're doing it. If it serves a need/ if it kind of serves a need and you don't have other work to do it, that's fine. If it's something that you think would be fun to do but could be solved with a 3rd party option (which theoretically already has most major bugs cleaned out of it), then it's a bad idea. – Misjudge

Tom, thanks for your careful consideration. My number one objective is that whatever I do is feature-rich, low on bugs, plays nice with others, and can be easily expanded on in the future. That pretty much points like a compass to using a third-party app / service over something I build myself. Also, in general, I lean towards cloud-type services for this sort of thing so I don't have to worry about the minutiae. Let the experts at MailChimp, etc, worry about the best techniques for not getting flagged as spam, or for tracking opens / clicks. – Drawl

If you've got a properly configured MTA then this will actually slow down the mail processing - queueing mail is the function of the MTA and it should be much more efficient to do it there than in your own code. – Tzar

Answer 4

Use Google AppEngine if you are worried about scalability & customization: it provides an email API and you can interface anything to it provided it is through an HTTP interface.

Of course, this is just a suggestion - disregard if this doesn't fit.

Matchboard

Answer 5

Why not keep your PHP system and use a third party SMTP service? Some reliable companies offer the use of e-mailing-only machines at reasonable prices, e.g. Dewahost who I am planning to use.

Also see the question Third Party Email Senders and my answer there.

Overseer

Answer 6

Quite possibly not ideal, but if you're looking at large scale transmission there are commercial solutions such as Port 25's PowerMTA that can be set up to effectively transmit the contents of a given folder.

As such, you'd simply use PHP to create the personalised MIME formatted raw data for each outbound email and place these in a temporary directory prior to transmission. (I've written such a system in the past and you'd be surprised at the PHP's ability to grind out the emails, even with quite complex text & HTML emails with images as inline-attachments, etc.) Once you were ready to transmit, you'd then move the files en-masse to the PowerMTA monitored folder and it would take care of the transmission.

Depending on how you look at it the benefit/problem with such a solution is that you'll need to build trust relationships with people such as AOL, MSN/Hotmail, etc. to ensure that your mail server isn't blacklisted due to user's reporting email as SPAM. (That said, this will likely be a factor with any DIY solution.)

Beirut

Answer 7

Check out Campaign Enterprise for a possible in-house solution.

Gambill

Answer 8

One of my friend uses http://www.tailoredmail.com but i havn't used it personally

Firsthand

Answer 9

I know this is an older question, but I just wanted to suggest SendGrid which is essentially an "Email Server as a Service" allowing you to send emails with cost per email.

Authorization