How to implement MapReduce in C# using PLINQ?

Question

How to implement MapReduce in C# using PLINQ?

Suppose, you have 7-8 WebServices to collect data and on each receiving (async manner) you have to put that data into some tables of a database, in my case it is SQL Server 2008. For instance the data you are getting from each Web Service is:

 <employees>
  <employee>
   <name>Ramiz</name>
  </employee>
  <employee>
   <name>Aamir</name>
  </employee>
  <employee>
   <name>Zubair</name>
  </employee>
</employees>

And, on each receiving of response this data goes into a table name - Employee:

Employee
===
EmployeeID (PK)
EmployeeName

Once the data goes into table, it has to return as json to the client which is ASP.NET (MVC 3) application is making this call using client-side JavaScript (ajax).

Suppose, a WebServiceEmployee1 has returned with data and other 6 are in queue (still trying to get the data). Then, it should goes register the resultset into table instead of waiting other 6 and return data of inserted employee to client in json. And, keep it connected and doing while others do the same way.

Please see, in my toolbelt I have ASP.NET MVC 3 (Razor), SQL SERVER 2008 R2, jQuery.

Thanks.

Howlet

Pleas give us a simple example of data you are going to process and the processing result you expect to get at the end. – Longways

I deleted my answer because i don't quite understand your context and how you can use MapReduce where with or without linq – Unemployment

Thank you for explaining your scenario. I am still missing following points: 1. What is your current solution? 2. Why aren't you satisfied with your solution? What are you hoping to improve using PLINQ? 3. Why do you need PLINQ at all? – Longways

@achitaka-san my contemporary solution is parallel looping (Parallel.ForEach) and making request every service and storing the resultant in a list. once, that loop ends i forward the call to do the database work inserting items from list into employee table. and, finally send that list as json to client. the problem, all this process is get problematic if a service delay acquiring records and the client (ajax) fall in timeout. also, this gives client a feel of a long delay. – Howlet

See Map / Reduce – A visual explanation – Athalla

Answer 1

Without clear understanding on the processing you want to perform, I suggest reading though the following MSDN documentation: http://bit.ly/Ir7Nvk It describes map/reduce with PLINQ with examples like:

public IDMultisetItemList PotentialFriendsPLinq(SubscriberID id, 
                                           int maxCandidates)
{
  var candidates = 
    subscribers[id].Friends.AsParallel()                    
    .SelectMany(friend => subscribers[friend].Friends)
    .Where(foaf => foaf != id && 
           !(subscribers[id].Friends.Contains(foaf)))
    .GroupBy(foaf => foaf)                               
    .Select(foafGroup => new IDMultisetItem(foafGroup.Key,
                                       foafGroup.Count()));
  return Multiset.MostNumerous(candidates, maxCandidates);
}

The "map" being Friends.AsParallel, SelectMany, and Where and the "reduce" phase is the GroupBy and Select

Vanhook

Answer 2

This PDF from MS has a Map Reduce example towards the middle/end

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CH0QFjAE&url=http%3A%2F%2Fdownload.microsoft.com%2Fdownload%2F3%2F4%2FD%2F34D13993-2132-4E04-AE48-53D3150057BD%2FPatterns_of_Parallel_Programming_CSharp.pdf&ei=4f_VT-ScD-Lg2gWqoeSWDw&usg=AFQjCNGhk_BZL8-5n8DaS_kMTaWRU9Y1Zw&sig2=ddKl4KuOGUiUb1pIawWeNQ

public static IEnumerable<TResult> MapReduce<TSource, TMapped, TKey, TResult>(
this IEnumerable<TSource> source,
Patterns of Parallel Programming Page 76
Func<TSource, IEnumerable<TMapped>> map,
Func<TMapped, TKey> keySelector,
Func<IGrouping<TKey, TMapped>, IEnumerable<TResult>> reduce)
{
return source.SelectMany(map)
.GroupBy(keySelector)
.SelectMany(reduce);
}

Suspensory