Actually user code can do anything. it would be hard to tackle special cases. In my opinion the best way to go is:
a) create sandbox appdomain with execution permission only. This ensures a lot of things like inability to mess with file system or make calls to native libraries.
b) create new process and start your appdomain in it.
Then monitor process tightly for memory and cpu consumption. If anything goes wrong - kill it. Note that it's the process you can kill, not appdomain. With appdomain you can try to unload it, but if malicious code running in finally clause then it wouldn't work.
There are still some (known to me) issues with this:
- Be careful where you place user compiled assemblies. Actually even in least privileged appdomain user code will be able to load assemblies that are in the same directory and execute them.
- I mentioned you should monitor process tightly. Code (running in finally clause) which spawns threads in infinite loop, where each thread does the same grabs memory extremely fast (at my observations). if an attacker decides to make dos attack with such code - who knows what happens:) Perhaps one way to leverage this is to start user process with low priority so that supervising threads have a chance of proper monitoring in a loaded system.
So all in all there is a risk anyway. i was toying around with this idea too and here is the current result: rundotnet