Message queues in ASP.Net Web API

Question

I am developing a client-side single-page-application (SPA) with AngularJS and ASP.Net WebAPI.

One of the features of the SPA includes uploading large CSV file, processing it on the server, and returning the output to the user.

Obviously, this kind of computation can not be done online, and therefore I implemented an UploadController in charge of receiving the file, and a PollingController in charge of notifying the user when the computation is complete.

The client side application monitors the PollingController every few seconds.

I have no experience in Message Queues, but my gut tells me that they are required in this situation.

How would you recommend to implement this functionality in a non-blocking, efficient way ?

Examples will be highly appreciated

Evenings

Answer 1

I've used message based service bus frameworks for this in the past.

You write an application (running as a windows service), that listens for messages broadcast across a event bus.

Your frontend can publish these messages into the bus.

The most popular framework for this in .NET is NServiceBus, however it recently became commercial. You can also look into MassTransit, though this one has very poor documentation.

The workflow you would do:

• MVC App accepts upload and places it into some directory accessible by the windows service
• MVC App publishes "UploadReady" message.
• Service receives message, processes file, and updates some state for the polling controller.
• Polling controller watches for this state to change. Usually a DB record etc.

The nice bit about using a framework like this is that if your service goes down, or you redeploy it, any processing can queue and resume, so you won't have any downtime.

Alpert

What are the advantages of using NSeviceBus/MassTransit over using the MSMQ directly ? – Evenings

They abstract away the backend transport and let you just focus on the business logic, they also add fault tolerance, retries, etc. – Alpert

Just checked out the documentation for Mass Transit and it has improved a lot since this post was written so it might be worth your time. – Indiction

Answer 2

For long running operations you need separate Windows Service application (or Worker Role, if it is Windows Azure). IIS may kill ASP.NET processes on pool recycling and your operation will not finish.

Message queue is mostly for communication. You can use it between your web and worker parts. But it is not required there unless your data is not super critical. You can establish communication using database, cache, file system or 100 other different ways :)

You can use SignalR to notify your client about finished processing.

Nihilism