Framework recommendation needed on .NET or JAVA for Volunteer computing (internet nodes)

Question

I want client machines on internet who subscribe to my server to donate their idle cpu cycles.(Like SETI@Home)

They would take jobs(work-units) from server to process, and send back results to the server. (This is the most simple description). The framework i need should allow me to define a job/task. Rest of things like communication, job execution/tracking, client binaries update etc. should be managed by framework.

• I evaluated Alchemi.NET a bit, but its not actively maintained, seems half-baked.

• BOINC has API in C, but i want a .NET or JAVA framework.

• I am looking at Manjrasoft's ANEKA , but it seems to work only for LAN clouds.

There must be some such frameworks available. I need expert recommendations!

Superintendent

Answer 1

I'm hardly an expert but I do have a little experience with distributed computing using MPI (with C). What you're talking about does not sound like grid computing, rather a master/slave system. That is your server is the master and it directs all the clients (slaves).

I know very little about .net programming so I'll speak in general.

There are a lot of web frame works out there and probably most have the facilities you'll need. That is your client will be able to upload files with the content they have gathered (or they could just use http get/post), because you don't have to worry about UI issues you can handle everything probably through one action (assuming an action based web framework). Then the server can return a response via JSON or XML which the client can use to take further direction on. JSON is the right choice if the system is very simple and probably a good choice for prototyping.

For the application upgrade I would consider this a separate issue (although it should be a simple matter for the server to return this to the client).

Rowenarowland

Answer 2

BOINC is the framework that most naturally meets your volunteer computing requirements, and it's stable and highly scalable -- I'd make sure you've considered it completely before ruling it out.

If you need to deliver something to a short deadline then I'd consider working up a simple supervisor (or scheduler) - worker pattern, just to get the application off the ground. The supervisor would be responsible for chunking up the data and making it available over http. Workers (your client app) would connect to a supervisor server; download a chunk of work; complete the chunk; and upload the results to the supervisor.

The main trick is to get the state machine thrashed out properly, so that you can accurately track what state each work chunk is in. I'd have the supervisor persist state in a database in the background.

Does your first release need to be internal, or is it for public consumption?

While that's working I'd get started on looking at getting up to speed with BOINC and planning a migration.

Spina

Hello Tim :) I need to get the business logic working first. I also am fairly comfortable with little home-grown solution (supervisor-agent as you pointed), Infact i have the basic design clear and some of it is coded as well. But i dont want to invest in that only to realize later that existing framework would have fitted and more. Obviously i will have to deal with all the problems as they come (of my written framework) and solve them incrementally, which already have been solved well by existing FX. – Superintendent

But i am new to Distrib. computing, so i dont know how much time will it take to understand existing FXs and find best and then try to use them. – Superintendent

BOINC's API is in C. I havent investigated it fully, but i get a sense that it will be an effort (and to determine whether really it is right approach) to get in place a .NET wrapper first. And to get it working etc. – Superintendent

We have basic http get/post libraries, XML parsing infrastructure and related utilities in place (many of other raw material utilities are freely available also on internet). So that way constructing basic path shouldnt take much time, but you know the intricate agent and its job management logic will have unforseen problems to be solved. So question is ,should i invest bootstrapping time today to select an existing FX OR rule out that option(if none fits), or shudi go my own and keep learning more (in background looking out for suitable FX) and concentrate to get business logic working first. – Superintendent

Apologies -- tied up with work, will try and take a look at the comments later today :) – Spina

Sure, even as it is night time in my zone. Only that the answer and discussion to this issue will be really important for me. Thanks :) – Superintendent

@Tim Could you get chance to read additional info in my comments ? – Superintendent

Answer 3

My recommendation

Work dist:

1. Have a receiver of requests, that places messages in a message queue, like rabbit mq
2. Have a host of workers, listening to the same queue, taking work from it and acking it when done.
3. When done, send a message on another queue, containing an URI to a known location, such as your network drive. The target is your parsed data.
4. The receiver listens to these "completed" messages. Fetches the data from the URI.

Done.

RabbitMQ comes with great CLR APIs.

The same reasoning works well with Microsoft Azure and their AbbFabric Queue. A plus is that is scales extremely well.

Hot Versioning

http://topshelf-project.com/

It gives a folder where you can drop binaries, which are then run. It manages versioning of these as well as running them as windows services.

Deployment

You can deploy the binaries with robocopy/xcopy and "net use Q: pwd \server\share", "net delete Q:"

Continuous Integration

Teamcity

After working with MsBuild extensively, I would recommend scripting it with psake and running the build with PowerShell. If you get advanced with PowerShell you also have WinRM available to you from your build scripts, which is really smooth.

Use the git/subversion commit number as the 0.0.0.x, x in the previous version number, and you will have automatic versioning that is "shared" across "Debug"/"Production" builds.

The Azure way

Work dist:

Same as above but with AppFabric Queue instead of RabbitMQ.

Hot Versioning

By swapping "Staging" and "Production" instances around, you avoid the downtime.

Deployment

You can either tap into the Azure Tools for Visual Studio's MsBuild tasks as can be read about here or you could use the PowerShell AzureSnapIns with a similar setup as above for Continuous Integration.

Continuous Integration

Same as above.

Kendo

Answer 4

How about .net's ClickOnce installer to manage auto updated client binaries. http://msdn.microsoft.com/en-us/library/t71a733d.aspx

I'm not sure of a "jobs framework" per-say, but Microsoft's Sync framework to support rolling your own jobs syncing with clients?

http://msdn.microsoft.com/en-us/sync/default

Dimple