Implementing C# language extensions

Question

Using systems such as Parallel Linq, it's possible to split up execution of anonymous functions, queries, etc across multiple cores and threads within a single machine. I'd like the ability to extend this to run across multiple machines using standard language constructs such as for loops (like Parallel.For()), value types like ints, structs, etc., and keep the application source modifications to a minimum. Ideally, this would allow me to open a project, add an attribute to a method, and recompile to gain access to the enhanced functionality.

It seems that I'd need something along the lines of:

1. The ability to capture a compiled block of code (such as a lambda) and pass it to a worker process running on another node, along with any data that is required, or

2. Provide a preprocessor that would capture the code in question, compile it in a sort of template project that would replace variable references, etc, with references to a class that would handle network communication, caching and access to any other required assets, and send the resulting DLL to any available worker nodes running on other machines.

Roslyn appears to provide some tools that would be useful here. Is there way to hook into the current compilation pipeline to allow this?

Edit

Okay, I know this is possible, because these guys did it. The question is, how?

Liberate

Okay, closers - How exactly is this too localized? – Liberate

Because you must be the only person here who wants to implement C# language extensions for creating OpenCL kernels. – Megdal

Okay, forget about the OpenCL part. How would you go about hooking into the compilation phase to customize code generation for distributing execution across heterogenous platforms? That hardly seems more clear to me. And since when do we close questions about specific applications?! – Liberate

And besides, only Microsoft can extend the C# compiler. – Megdal

@DavidLively var isTooLocalized = Math.Pow(obscureTechCount, 2) > 16; if (isTooLocalized) { VoteToClose(); } – Compound

I don't think this is that localized. All the OP is really asking for is guidance on writing a LINQ query provider to transform C# to OpenCL which is a large undertaking but not terribly complicated up front. See here – Vanguard

@JohnSaunders fine. See the update. – Liberate

Maybe I misinterpreted your request for impossible language extensions. If, instead, you asked for help creating a LINQ provider, then I would not have voted to close. How about fixing your title to reflect what you're asking for. And, BTW, no, there's no way to hook the compilation pipeline pre-Roslyn. – Megdal

@RonWarholic thanks very much - that's pretty close to what I'm looking for. – Liberate

@JohnSaunders then "There's no way to do that" is a valid response, since I was asking IF it's possible, and IF SO, HOW. I don't see how that makes the question "too localized." – Liberate

But perhaps you don't need to hook the compilation pipeline. A DSL can generate code through a T4 template, and that generated code can then be compiled through the normal compiler. – Megdal

@JohnSaunders ... which would also be a valid answer, though it really falls under the preprocessor category. I'm not sure why you didn't post an answer. – Liberate

Because I wasn't sure what you wanted. Your question has changed substantially. Also, haven't the GPU.NET people documented how they did it? – Megdal

Note that you are now getting better answers because you've changed from asking how to do something impossible to telling us what you're trying to achieve and asking us how you can achieve it. There's a lesson there. And I still wish you'd change your title to reflect what you're actually asking. – Megdal

Answer 1

You don't have to extend the language persay to do what Brahma does. He just implemented a custom query provider that parses expression trees and emits GPGPU code (LINQ to SQL does the same thing but with SQL).

I linked a basic guide on MSDN here that can get you up and running implementing an IQueryable provider.

The hard part will be traversing the expression trees and generating OpenCL code. Once you can do that you just pass it off to Cloo and you should be running.

Edit

You linked a tool that compiles standard .NET code to GPU code with an attribute [Kernel]. They do this by having a post-build tool look for the attributes in the compiled IL and they perform IL rewriting to generate GPU calls. This is similar to PostSharp, an AOP solution.

IL rewriting is time consuming and hard work but you could also go this route.

Vanguard

Awesome; thanks for the response. I'm still curious about implementing this without having to use a custom provider - such as capturing loops, etc and replacing them with references to such a provider - but this will solve the problem. – Liberate

+1. There are some libraries that purport to make implementing a custom LINQ provider easier, such as re-linq. – Baeda

Expression trees can represent loops and most statement constructs as of C# 4. If you don't want the LINQ style syntax just parse the expression trees yourself. Remember that you can assign lambda functions directly to expression objects, eg: Expression<Func<int,string>> someExpression = (i) => return "Hey!" – Vanguard

I've seen references to expression trees but haven't investigated them. That seems promising. – Liberate

Question edited to appease the naysayers; your original answer still applies. I'd change the question back but id probably be struck by lightning. – Liberate

Answer 2

Using systems such as Parallel Linq, it's possible to split up execution of anonymous functions, queries, etc across multiple cores and threads within a single machine. I'd like the ability to extend this to run across multiple machines using standard language constructs such as for loops (like Parallel.For()), value types like ints, structs, etc., and keep the application source modifications to a minimum.

Sounds great. In fact we have a system very much like that over in Microsoft Research, though obviously I cannot discuss the details.

I need the ability to capture a compiled block of code (such as a lambda) and pass it to a worker process running on another node, along with any data that is required

OK, you've got it. We added that feature to C# 3. That's how LINQ to SQL works. Somehow the LINQ query has to get onto the database. The compiled lambda is interrogated on the client machine, transformed into a query which is sent to the server node, and then the result is sent back.

Roslyn appears to provide some tools that would be useful here. Is there way to hook into the current compilation pipeline to allow this?

That's not the purpose of Roslyn; Roslyn is not about adding new features to the C# language. It's about making it easier to analyze code to build things like refactoring engines.

You don't need to hook into the compilation pipeline. PLINQ doesn't change the compiler, LINQ to SQL doesn't change the compiler, and so on. When you convert a lambda to an expression tree the compiler emits code that creates an expression tree at runtime that represents the lambda. You can interrogate that expression tree, serialize it across to another machine in your network, deserialize it, turn it into a delegate and run it if that's the kind of thing you enjoy doing.

You'll need to write your own expression tree serializer and deserializer probably, but they are pretty straightforward data structures. Being an immutable tree should make them pretty easy to serialize and deserialize; they can't really form complex networks since they are always constructed from leaf nodes up.

Jauch

Thanks, Eric. I hadn't considered the similarity to LINQ to SQL. I've started playing with expression trees this morning, and am trying not to cackle too loudly. :) – Liberate

Follow up: this works amazingly well. Kudos to MS for providing such a flexible, extensible system. Providing compile -time validation of code-as-data is mindblowingly cool. – Liberate