Queue-Based Background Processing in ASP.NET MVC Web Application [closed]

Question

How can I implement background processing queues in my ASP.NET MVC web app? While most data changes, updates etc. need to be visible immediately, there are other updates that don't need real time processing which I would like to hand off to a lower-priority background process which will take care of it at its own pace.

As an example, take StackOverflow's badge award system. Usually you may take a specific action that would award you a badge, but the actual 'award' happens later (typically between 10 minutes and a couple hours later). I assume this is done via a separate background process as it is not critical for SO's workings to award badges immediately when earned.

So, I'm trying to create some kind of queue system in which I could stuff tasks (say anything implementing ITask interface which will have a Process() method) which will eventually get executed by a separate process.

How would I go about implementing such a system? Ideas/Hint/Sample Code?

Thank you!

Jurel

Answer 1

Windows Services and MSMQ to communicate with them (if you even need to).

-- Edit

To slightly expand.

You'll create several services, depending on what you want to do, and have them all run an endless thread of various sleeping levels, to do what you want. They will then update the database appropriately, and you will not have had to do anything on the client side.

You may wish to interact with them from an admin point of view, hence you may have an MSMQ that they listen to admin commands on. Depending on your implementation, you may need to restart them for some reason, or possibly just 'force' a running of whatever they wish to do.

So you'll use an MSMQ Private Queue to do it (System.Messaging namespace). One of the main things to note about MSMQ, is that message need to be < 4meg. So if you intend to send large object graphs, serialise to a file first and just send the filename.

MSMQ is quite beautiful. You can send based on a 'Correlation ID' if you need to, but, for some amusing reason, the correlation ID must be in the form of:

{guid}\1

Anything else does not work (at least in version 2.0 of the framework, the code may have changed).

-- Edit

Example, as requested:

using System.Messaging;

...


MessageQueue queue = new MessageQueue(".\\Private$\\yourqueue");
queue.Formatter = new BinaryMessageFormatter();

Message m = new Message();
m.Body = "your serialisable object or just plain string";

queue.Send(m);


// on the other side

MessageQueue queue = new MessageQueue(".\\Private$\\yourqueue");
queue.Formatter = new BinaryMessageFormatter();

Message m = queue.Receive();

string s = m.Body as string;

// s contains that string now

Overweight

Do you have an example on how to initially 'connect' a service application with a MSMQ and then to ASP.NET MVC? This is pretty new to me. Thank you! – Jurel

You just write to the appropriate queue. Your service will call .Receive and your page, that you want to send a message to the queue from, will simply call '.Send'. I will update with a small example. – Overweight

Updated. Note I haven't shown the code for creating the queue, if it doesn't exist. But that can be trivially found just by takinga like at the System.Messaging API. Specifically: MessageQueue.Create( ... ). Note you'll also need to get appropriate permissions. (.SetPermissions). – Overweight

Calling queue.Send(m) is completely asynchronous correct? How would I communicate back to the MVC application/how do I get a notification that something is waiting in the queue to get processed? Thanks again! – Jurel

queue.Send will return immediately, yes. You communicate back via another queue. You can simply wait for messages, or as I hinted earlier, use CorrelationId to ensure you get a response for the message you sent (queue.MessageReadPropertyFilter.CorrelationId before calling .Receive(). You can call .Receive() with a TimeSpan for a timeout, so you may do that, and note that you haven't got anything back yet, so processing isn't done, on the queue's side. – Overweight

How do I wait for messages? That part isn't clear to me yet. Also, you have Message m = queue.Receive() in your example, but what happens when there is more than one message 'waiting' ? – Jurel

calling .Receive() blocks until a message is received. When a message is placed on the queue, it returns, and you can do your work. Thus, as I said earlier, you have this part of your code in a while(true) loop in a separate thread. So it will run forever, waiting for messages, and as soon as it picks one up, it will (probably) get a thread from a thread pool, and send the message of for processing, then go back to waiting for messages. This answers the second part of your question; the message will wait until it's picked up. – Overweight

@NoonSilk, how could I use this to have a user start a long running process (say a few seconds) and then see the results in the browser ASAP once the task has been processed by a worker on a pool of servers? – Brassy

Answer 2

Jeff has a great post showing how he achieved this originally for Stack Overflow at https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/

While it might not be as reliable as a service if that's an option to you, it's served me well in the past for non essential tasks, and I've had it working in a virtual hosting environment.

Slocum

Answer 3

Just found this question when searching for background process in ASP.NET MVC. (Available after .NET 4.5.2)

public ActionResult InitiateLongRunningProcess(Emails emails)
{
    if (ModelState.IsValid)
    {
       HostingEnvironment.QueueBackgroundWorkItem(ct => LongRunningProcessAsync(emails.Email));
       return RedirectToAction("Index", "Home");
    }

    return View(user);
}

Note: Personally I won't use the Webserver to run background tasks. Also don't reinvent the wheel, I would highly recommend to use Hangfire.

Read this great article from Hanselman HowToRunBackgroundTasksInASPNET

Integrand