What I do is process and store the results on the server in a background thread that reads from a message queue. In our case, this is for insurance quotes. As each carrier's insurance quote is finished, its premium is stored in our database.
On the client side, we call the server using a timer loop. See Ben Alman - doTimeout
You can see this in action at Autoquoter.com. Just use an illinois zip code and enter some dummy info to get a quote (zipcode=60010 for example). You can view the source for our client side code.
Here is our server side code. Each message is an xml serialized insurance carrier.
var queue = new QueueManagerSoapClient("QueueManagerSoap");
var messages = queue.RemoveAll(session.Id);
if (messages.Any())
{
var serial = new XmlSerializer(typeof(Carrier));
foreach (var message in messages)
{
var carrier = serial.Deserialize(new StringReader(message)) as Carrier;
found = carrier != null;
if (found)
{
session.Carriers.Add(carrier);
}
}
session.SaveChanges();
}
return View("ResultView",session.Carriers);
UPDATED:
There are three components to make this work.
First you'll need to launch an asynchronous request when your web page is loaded. This request would load a page that performs your search. It should store results into your database as they are received, or add them to a message queue service. You'll need to mark the search as completed.
<%
var wc = new WebClient();
wc.DownloadStringAync(new Uri("http://baseurl/search/submitrequest"));
%>
Secondly, add a timer to your web page that performs an ajax load periodically (every 10 seconds for example)
var interval = setInterval(function() {
$('#resultsDiv').load("/search/showresults");
},10000);
Lastly, you'll need to terminate the interval when all of the results have been stored in the database. You could add something like this at the bottom of the search results page.
<% if ( isFinished ) { %>
<script type="text/javascript">
interval.stop();
</script>
<% } %>