Sending out 20,000+ emails with asp.net

Question

I am writing an application that will need to send a massive amount of emails to our students who will be selected from our database (each email will be personalized to the extent that will include their name, course of study etc...so needs to be sent one at a time).

I could do this looping over an SmtpClient, but I'm afraid that with the numbers I'm trying to send, I'll ultimately run into timeout issues or my thread being killed because of lack of machine resources.

At this point I'm just looking for suggestions of a better way to handle this, or if looping over SmtpClient is an ok solution, how I should go about handling it to prevent what I posted above.

Would a web service be a better alternative?

Please advise, TIA

Dilatant

you need to go with some third party tools – Dope

you do NOT need to go with some third party tools – Bischoff

Answer 1

My suggestion would be to batch this out. Do not try to run ASP.NET code to create 20K emails and send them out as you are bound to runinto timeout and performance issues. Instead populate a table with recipient, subject, body and begin a batch process from a windows service. This way your code is executed in a way where lag can be managed, instead of having a web page wait for the request to return.

Incomplete

this approach sounds promising. I'm already saving the emails in a table. Sorry for a noobish question, but how would I call this windows service to have it initiate the sending of the emails, and likewise, how would I report back to the client which ones have been sent successfully and which ones failed from the windows service? – Dilatant

you wouldnt have to call the windows service. just let it run. The service could be designed to check your table and see if there is any work to be done. if there are emails that have not been sent then batch them up and send them. Basically the service should be designed to query that table ever 5 min or so. (Note you should put a bool flag on that table saying if it has been sent and for preformance purposes put an index on the bool flag. ) Now when you query the table for all items not sent, if count is greater than one then the service begins emailing. As for reporting back to the client – Incomplete

my last comment got cut off. As for reporting back to the client you could diplay your log on a web page stating weather or not an email has been sent. This display should be divide into two views. sent and unsent. (Careful with this one as it could be a long load with 20K records). – Incomplete

one more quick question about the windows service. How can I break it up so that it's not trying to send all 20k+ emails at one time? I think I may ultimately run into the same or similar problem unless I have it doing it in "chunks" as someone else suggested here.... – Dilatant

your question is quickly by the beauty of this design. Because you are only queuing up the emails. you can adjust the query the service runs ever 5 mins to SELECT Top 100 * FROM MyMailTable where notsent = true. This way every 5 min or whatever interval you create the service will send out 100 emails. – Incomplete

Answer 2

Ok, first - this is hardly massive, I have been handling 50.000 emails+

Let the emails be written to a FOLDER - directory. Then use the local SMTP service you can install, pointing it at the folder as pickup directory. This will at least make sure you have a decent buffer in between (i.e. you can finish the asp.net side, while the emails are not sent at that point).

Henze

well first issue that comes to mind is that all of our students use a "3rd party" for their email (all using the same one), that has our exchange server white-listed as "an ok sender". Unfortunately I don't have access to the exchange server to tell it to fetch the emails from this 'folder'...that's all on our infrastructure's shoulders. I also don't have access to add the web server as an "ok sender" on our 3rd party (where all of the email addresses are)...so can't use it as an smtp server there. If I created a web service to handle sending the emails through our exchange...would that fly – Dilatant

Actually where is the problem? Seriously... The LOCAL SMTP service can be configured to (a) use the exchange server as relay and (b) authenticate there using a username and password, as you would with your normal smtp service. – Henze

your answer does not provide any solution whatsoever, instead, you're just asking more question. Did you expect 20,000 users to all be on the same email provider? Your answer shows that you have no experience in sending this sort of emails. – Tephrite

@Helen - this is retarded. Sorry. An email server using a relay will send all emails through one server like ANY OTHER ONE. THis is how it works. He has the same sender for all 20.000 emails, and the EMAIL SERVER will find out where to send things. If you do not know how email works, Helen, I suggest you get up and read the basics FIRST, instead of posting a foolish comment. ANY email server (the exchange server in this case) is able to find out where to send an email. – Henze

Yes, this is how we do it, using the PickupDirectory feature of SmtpClient. Directory gets filled with e-mails named by GUIDs. Then copy them over to the "real" pickup directory that Exchange is using (we use the temporary directory in case the generation process bombs halfway through, we haven't sent out some e-mails by mistake and can just delete and restart), then Exchange sucks them up--no worries about overloading, Exchange is smart enough to only suck up a few at a time (though this is configurable). – Wendell

Answer 3

What about using Database Mail tool provided by SQL Server itself. You can still use asp.net but Email will be sent by SQL Server and all you have to do call the stored procedure and leave things to SQL Server.

I wouldn't recommend batch option since there won't be any optimization or queue thing unless you have to put a lot of effort to do so.

Database Mail tool provides queue and priority options as well. If a problem occurs, the e-mail will be queued again and sent later but the other ones will be still being sent.

It is called Database Mail in SQL Server 2008 and SQL Mail in previous versions.

For more information, please check the links below :

• http://blog.sqlauthority.com/2008/08/23/sql-server-2008-configure-database-mail-send-email-from-sql-database/
• http://msdn.microsoft.com/en-us/library/ms175887.aspx

Nf

THat thing simply sucks and should never nave been in SQL Server to start with. – Henze

please ignore TomTom - he was born drunk! Have you seen some of his other comments? – Tephrite

I can agree that a mail sending program should probably not be part of a database server, but the fact remains that it is. This could be a good solution for the OP. – Tolley

Answer 4

You could use javascript on a timer to request the script which sends mail in small chunks that wont time out. Then you'd call the script from the browser. Add a progress bar, authentication, etc.

Cognition

Answer 5

You don't need a web service, but rather a way to batch the emails into a queue (perhaps an "Email_Queue" table in the DB). Then a Windows service running on a server could read through the queue in a first-in/first-out basis at reasonable chunks at a time being sent to the SMTPClient, removing items from the queue as they are processed. You would probably need to run some measurements to determine what the chunk size and delay would be for your mail server.

Cephalonia