Recompile C# while running, without AppDomains

Question

Let’s say that I have two C# applications - game.exe (XNA, needs to support Xbox 360) and editor.exe (XNA hosted in WinForms) - they both share an engine.dll assembly that does the vast majority of the work.

Now let’s say that I want to add some kind of C#-based scripting (it’s not quite "scripting" but I’ll call it that). Each level gets its own class inherited from a base class (we’ll call it LevelController).

These are the important constraints for these scripts:

1. They need to be real, compiled C# code

2. They should require minimal manual "glue" work, if any

3. They must run in the same AppDomain as everything else

For the game - this is pretty straight forward: All the script classes can be compiled into an assembly (say, levels.dll) and the individual classes can be instanced using reflection as needed.

The editor is much harder. The editor has the ability to "play the game" within the editor window, and then reset everything back to where it started (which is why the editor needs to know about these scripts in the first place).

What I am trying to achieve is basically a "reload script" button in the editor that will recompile and load the script class associated with the level being edited and, when the user presses the "play" button, create an instance of the most recently compiled script.

The upshot of which will be a rapid edit-test workflow within the editor (instead of the alternative - which is to save the level, close the editor, recompile the solution, launch the editor, load the level, test).

---

Now I think I have worked out a potential way to achieve this - which itself leads to a number of questions (given below):

1. Compile the collection of .cs files required for a given level (or, if need be, the whole levels.dll project) into a temporary, unique-named assembly. That assembly will need to reference engine.dll. How to invoke the compiler this way at runtime? How to get it to output such an assembly (and can I do it in memory)?

2. Load the new assembly. Will it matter that I am loading classes with the same name into the same process? (I am under the impression that the names are qualified by assembly name?)

   Now, as I mentioned, I can’t use AppDomains. But, on the other hand, I don’t mind leaking old versions of script classes, so the ability to unload isn’t important. Unless it is? I’m assuming that loading maybe a few hundred assemblies is feasible.

3. When playing the level, instance the class that is inherited from LevelController from the specific assembly that was just loaded. How to do this?

And finally:

Is this a sensible approach? Could it be done a better way?

---

UPDATE: These days I use a far simpler approach to solve the underlying problem.

Mantle

Why is it that you must avoid AppDomains? – Televise

The "script" needs to manipulate objects that must be in the default AppDomain (because of XNA). Also performance is an issue (at least for game.exe). – Mantle

You can't unload assemblies from an AppDomain, so you're probably going to end up loading tons of "temp" assemblies into your AppDomain. This could become quite confusing to work with, as the namespace/class names will presumably be the same! – Felder

@DanTup Not really - classes are identified by assembly as well as name. And as long as you don't accidentally hold onto references to classes from the "old" assembly, after you start using the "new" assembly, it's not really an issue. – Mantle

Russel - It's certainly possible, just possibly "confusing to work with". Some errors (eg. InvalidCastExceptions) might not show assembly names, so you end end up with fun "Cannot cast Namespace1.Class1 to Namespace1.Class1" errors. Certainly possible though! – Felder

Answer 1

Check out the namespaces around Microsoft.CSharp.CSharpCodeProvider and System.CodeDom.Compiler.

Compile the collection of .cs files

Should be pretty straightforward like http://support.microsoft.com/kb/304655

Will it matter that I am loading classes with the same name into the same process?

Not at all. It's just names.

instance the class that is inherited from LevelController.

Load the assembly that you created something like Assembly.Load etc. Query the type you want to instanciate using reflection. Get the constructor and call it.

Seaside

Would I be right in saying that, to instance the type, after calling CodeDomProvider.CompileAssemblyFromFile, I would have something like: compilerResults.CompiledAssembly.GetType("MyLevelController").GetConstructor(...).Invoke(...); – Mantle

Yes, one way is like: Assembly generatedAssembly = Assembly.Load(fromMemory); OR Assembly generatedAssembly = Assembly.LoadFrom(assemblyFile); Type[] types = generatedAssembly.GetTypes(); object instance = types[0].GetConstructor(new Type[0]).Invoke(new object[0]); – Seaside

A shortcut might be "generatedAssembly.CreateInstance(typeName) – Seaside

CodeDom i s really a deprecated API, avoid using it is not without it's limitations. – Psychology

Ok, that's interesting news. But what is the appropriate, equally complex approach then? I mean what you wrote in your answer sounds a lot more complicated to me. – Seaside

Answer 2

There is now a rather elegant solution, made possible by (a) a new feature in .NET 4.0, and (b) Roslyn.

Collectible Assemblies

In .NET 4.0, you can specify AssemblyBuilderAccess.RunAndCollect when defining a dynamic assembly, which makes the dynamic assembly garbage collectible:

AssemblyBuilder ab = AppDomain.CurrentDomain.DefineDynamicAssembly(
    new AssemblyName("Foo"), AssemblyBuilderAccess.RunAndCollect);

With vanilla .NET 4.0, I think that you need to populate the dynamic assembly by writing methods in raw IL.

Roslyn

Enter Roslyn: Roslyn lets you compile raw C# code into a dynamic assembly. Here's an example, inspired by these two blog posts, updated to work with the latest Roslyn binaries:

using System;
using System.Reflection;
using System.Reflection.Emit;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace ConsoleApplication1
{
    public static class Program
    {
        private static Type CreateType()
        {
            SyntaxTree tree = SyntaxTree.ParseText(
                @"using System;

                namespace Foo
                {
                    public class Bar
                    {
                        public static void Test()
                        {
                            Console.WriteLine(""Hello World!"");
                        }
                    }
                }");

            var compilation = Compilation.Create("Hello")
                .WithOptions(new CompilationOptions(OutputKind.DynamicallyLinkedLibrary))
                .AddReferences(MetadataReference.CreateAssemblyReference("mscorlib"))
                .AddSyntaxTrees(tree);

            ModuleBuilder helloModuleBuilder = AppDomain.CurrentDomain
                .DefineDynamicAssembly(new AssemblyName("FooAssembly"), AssemblyBuilderAccess.RunAndCollect)
                .DefineDynamicModule("FooModule");
            var result = compilation.Emit(helloModuleBuilder);

            return helloModuleBuilder.GetType("Foo.Bar");
        }

        static void Main(string[] args)
        {
            Type fooType = CreateType();
            MethodInfo testMethod = fooType.GetMethod("Test");
            testMethod.Invoke(null, null);

            WeakReference weak = new WeakReference(fooType);

            fooType = null;
            testMethod = null;

            Console.WriteLine("type = " + weak.Target);
            GC.Collect();
            Console.WriteLine("type = " + weak.Target);

            Console.ReadKey();
        }
    }
}

In summary: with collectible assemblies and Roslyn, you can compile C# code into an assembly that can be loaded into an AppDomain, and then garbage collected (subject to a number of rules).

Complect

Just in case anyone else gets excited by Tim's answer here (as I did), please see Tomas' answer here. As of now, support for this method of collectible assembly creation was removed from the Roslyn CTP. – Sickness

@Sickness Oh, that's a pity! I see from a comment on the answer you linked to that there's a UserVoice suggestion to add the feature back in: visualstudio.uservoice.com/forums/121579-visual-studio/… – Complect

Answer 3

Well, you want to be able to edit things on the fly, right? that's your goal here isn't it?

When you compile assemblies and load them there's now way to unload them unless you unload your AppDomain.

You can load pre-compiled assemblies with the Assembly.Load method and then invoke the entry point through reflection.

I would consider the dynamic assembly approach. Where you through your current AppDomain say that you want to create a dynamic assembly. This is how the DLR (dynamic language runtime) works. With dynamic assemblies you can create types that implement some visible interface and call them through that. The back side of working with dynamic assemblies is that you have to provide correct IL yourself, you can't simply generate that with the built in .NET compiler, however, I bet the Mono project has a C# compiler implementation you might wanna check out. They already have a C# interpreter which reads in a C# source file and compiles that and executes it, and that's definitely handled through the System.Reflection.Emit API.

I'm not sure about the garbage collection here though, because when it comes to dynamic types I think the runtime doesn't release them because they can be referenced at any time. Only if the dynamic assembly itself is destroyed and no references exist to that assembly would it be reasonable to free that memory. If you're and re-generating a lot of code make sure that the memory is, at some point, collected by the GC.

Psychology

Answer 4

If the language was Java the answer would be to use JRebel. Since it isn't, the answer is to raise enough noise to show there's demand for this. It might require some sort of alternate CLR or 'c# engine project template' and VS IDE working in coordination etc.

I doubt there's a lot of scenarios where this is "must have" but there's many in which it would save lot of time as you could get away with less infrastructure and quicker turnaround for things that aren't going to be used for long. (Yeah there's some who argue to over-engineer things because they'll be used 20+ years but the problem with that is when you need to do some massive change, it'll likely be as expensive as rebuilding the whole thing from scratch. So it comes down to whether to spend money now or later. Since it's not know for sure the project will become business critical later and might require large changes later anyhow, the argument to me is to use "KISS"-principle and have the complexities of live-editing in the IDE,CLR/runtime and so forth instead of building it into each app where it might be useful later. Certainly some defensive programming and practises will be needed to modify some live service using this sort of feature. As Erlang devs are said to be doing)

Manvil