Dynamic Assembly Resolution/Management

Question

Short Version

I have an application which utilizes a plug-in infrastructure. The plug-ins have configurable properties that help them know how to do their job. The plug-ins are grouped into profiles to define how to complete a task, and the profiles are stored in XML files serialized by the DataContractSerializer. The problem is when reading the configuration files, the application deserializing has to have knowledge of all of the plug-ins defined in the configuration file. I'm looking for a way to handle the resolution of unknown plug-ins. See the proposed solution section below for a couple of the ideas I've looked into implementing, but I am open to just about anything (though I'd rather not have to reinvent the application).

---

Detail

Background

I've developed a sort of Business Process Automation System for internal use for the company I'm currently working for in C# 4. It makes exhaustive use of 'plug-ins' to define everything (from the tasks that are to be performed to the definition of units of work) and relies heavily on a dynamic configuration model which in turn relies on C# 4/DLR dynamic objects to fulfill jobs. It's a little heavy while executing because of its dynamic nature but it works consistently and performs well enough for our needs.

It includes a WinForms configuration UI that uses Reflection extensively to determine the configurable properties/fields of the plug-ins, as well as, the properties/fields that define each unit of work to be processed. The UI is also built on top of the BPA engine so it has a thorough understanding of the (loose) object model put in place that allows the engine to do its job, which, coincidentally, has led to several user experience improvements, such as, ad-hoc job execution and configure-time validation of user input. Again there is room for improvement, however, it seems to do its job.

The configuration UI utilizes the DataContractSerializer to serialize/deserialize the settings specified, so any plug-ins referenced by the configuration must be loaded before (or at the time of) configuration load.

Structure

The BPA engine is implemented as a shared assembly (DLL) which is referenced by the BPA service (a Windows Service), the Configuration UI (WinForms app), and a plug-in tester (Console application version of the Windows Service). Each of the three applications that reference the shared assembly only include the minimum amount of code necessary to perform their specific purpose. Additionally, all plug-ins must reference a very thin assembly which basically just defines the interface(s) that the plugin must implement.

Problem

Because of the extensibility model used in the application, there has always been a requirement that the config UI is run from the same directory (on the same PC) as the Service application. That way the UI always knows about all of the assemblies that the Service knows about so they can be deserialized without running into missing assemblies. Now that we are getting close to roll out of the system, a demand to allow the Configuration UI remotely on any PC in our network has come about from our network admins for security purposes. Typically this wouldn't be a problem if there was always a known set of assemblies to deploy, however, with the ability to extend the application using user built assemblies, there has to be a way to resolve the assemblies from which the plug-ins can be instantiated/used.

Proposed (potentially obvious) Solution

Add a WCF service to the Service application to allow the typical CRUD operations against the configurations which that instance of the service is aware of and rework the configuration UI to act more like SSMS with a Connect/Disconnect model. This doesn't really solve the problem so we would also need to expose some sort of ServiceContract from the Service application to allow querying of the assemblies it knows about/has access to. That's fine and fairly straight forward however the question arises, "When should the UI find out about the assemblies that the Service is aware of?" On connect we could send all of the assemblies from the Service to the UI to ensure that it always knows about all of the assemblies the service does but that gets messy with AppDomain management (potentially unnecessarily) and assembly version conflicts. So I suggested hooking into the AppDomain.AssemblyResolve/AppDomain.TypeResolve events to only download the assemblies that the client isn't aware of yet and only as needed. This doesn't necessarily cleanup the AppDomain management issues but it definitely helps address the version conflicts and related issues.

Question

If you've stuck with me this long I applaud and thank you, but now I'm finally getting to the actual question here. After months of research and finally coming to a conclusion I am wondering if anyone here has had to deal with a similar issue and how you dealt with the pitfalls and shortcomings? Is there a standard way of handling this that I have missed completely, or do you have any recommendations based on how you have seen this successfully handled in the past? Do you see any problems with the proposed approaches or can you offer an alternative?

I'm aware that not everyone lives in my head so please let me know if you need further clarification/explanation. Thanks!

Update

I've given MEF a fair shake and feel that it is too simplistic for my purposes. It's not that it couldn't be bent to handle the plug-in requirements of my application, the problem is doing so would be too cumbersome and dirty to make it feasible. It is a nice suggestion and it has a lot of potential, but in its current state it just isn't there yet.

Any other ideas or feedback on my proposed solutions?

Update

I don't know if the issue I'm encountering is just too localized, if I failed to properly describe what I am trying to achieve, or if this question is just too unreasonably long to be read in its entirety; but the few answers I've received have been subtly helpful enough to help me think through the problem differently and identify some shortcomings in what I am after.

In short, what I'm trying to do is take three applications which in their current state share information (configuration/assemblies) using a common directory structure, and try to make those applications work across a network with minimal impact on usability and architecture.

File shares seem like the obvious answer to this problem (as @SimonMourier proposed in the comments), but using them translates into lack of control and debugability when something goes wrong. I can see them as a viable short term solution, but long term they just don't seem feasible.

Interject

+1 For a well described Question! – Polyphonic

Sorry if it sounds stupid, but can't you just deploy your plugins assemblies centrally on a share and (provided the rights and CAS are ok) load them from this share, from any process (Service, Config, etc.)? Other question while I'm at it, are you aware of the cool but little known set of System.AddIn namespaces: msdn.microsoft.com/en-us/library/gg145020.aspx ? – Huntley

@SimonMourier - Thanks for the suggestions. Yes, I've heard of MAF but it is heavier than I was looking for when I wrote my own. As for deploying the plug-ins on a file share, I considered that for about 5 seconds and it could probably work but it just feels dirty. I think I'd prefer a more integrated solution (in fewer words: I'm a coder so I'd like to have to code something). – Interject

Is the problem "how can I dynamically load/unload assemblies?" or "how can I hook into the deserialization process?" Or have I missed it altogether? – Spell

@Spell - The issue is that the configs are serialized/deserialized using the DataContractSerializer and they contain serialized types that may not yet be known to the derserializing application. The question is how to handle this resolution. – Interject

Answer 1

tl;dr, but I'm 90% sure you should take a look into MEF.
When I first saw it I was like "aah, another acronym", but you'll see it's very simple, and it's built in into .NET 4. Best of all, it even runs seamlessly on mono and it's a matter of less than an hour (including coffee break) between hearing about it and compiling hello worlds to get used with the features. It's really that simple.

Basically, you "export" something in an assembly and "import" it into another (all via simple attribute decorations), and you choose where to search for it (example, on the applications directory, plug-ins folder, etc).

Edit: what if you try to download and load (and possibly cache) plugins on-the-fly on configuration load?

Needlecraft

The plugin structure is already well defined and from what I've read about MEF it wouldn't address the issue of making the plug-ins accessible/resolvable across the network. Have I overlooked this ability in the documentation somewhere? – Interject

Well, MEF uses catalogs, so you probably could subclass ComposablePartCatalog to make a "NetworkCatalog" that looks for assemblies in a network location (e.g., an FTP server). Or you could just download the assemblies at run-time? – Needlecraft

I'll look into the MEF catalog approach and possibly steal some concepts but even if I did switch the application to MEF I would have a lot of the same shortcomings just shinier ones. – Interject

Shinier shortcomings are the prettiest ones :) – Needlecraft

Thanks for the suggestion. I've updated my question based on my evaluation of MEF. – Interject

@Interject Too bad MEF did not meet your requirements, I really thought it would always hit the spot and that subclassing it to the extent of the features you wanted was feasible. What part of extending it was cumbersome? I'm curious. Also, I'll add an idea I had to the answer. – Needlecraft

In short, the part of MEF that would be cumbersome to implement is MEF itself. It provides a pretty way to identify how extension modules interface with a host application, which is great but the value it provides is nothing more than a glorified Type management API. I already have one of those and the one I built supports hot swapping of plug-in assemblies out of the box. Having to build a "NetworkCatalog" for it would be great if I had already been using MEF as my extension library, but if I have to write the logic to manage this anyway then where is the benefit of switching? – Interject

As for your suggestion to download and cache the assemblies on the fly; that is basically the approach taken in the proposed solutions. The question is targeted more towards the best way to implement it. – Interject

@Interject Hm, sorry for reading much less than I should, really. – Needlecraft

It's a long question, I'd bet at least half of the upvotes I've received for it are from people who haven't read it in its entirety. Like I said, MEF is a good solution for a lot of situations (and even mine if I hadn't already built my own) but at this point the amount of effort it would take to switch just isn't worth it for the little amount of benefit it would provide. – Interject

@Interject good to know that MEF would be your choice if you didn't have already written a (maybe better) alternative. Also good is to have nice people such as yourself here on SO, hope I grow up a little bit from the example. – Needlecraft

Answer 2

I think that you could be overlooking a relatively simple solution that derives somewhat from the Microsoft web.config approach:

Have two sections in the config file:

Section 1 contains enough information about the plugin (i.e. name, version) to allow you to load it into an app domain.

Section 2 contains the information serialized by the plugin.

On loading the plugin, pass the information in section 2 and let the plugin deserialize it according to its needs.

Snort

I can see how this would be beneficial but is there a way to achieve it using the DataContractSerializer to deserialize the config? – Interject

Yes. You have two choices here, I think: 1) Only deserialize the second section with DataContractSerializer. 2) Have a 2 pass deserialization. Pass 1 deserializes to a standard class that contains the plugin details in 1 property and a base64 encoded (for example) version of the plugin-serialized data. Pass 2 would be for the plugin to deserialize the data as appropriate. Hopefully that makes sense. – Snort

So basically, the steps are: 1: Use an XML parser to pre-parse the config to determine the distinct sections, 2: Deserialize the portion of the config describing the plug-ins referenced and resolve the references, and 3: Deserialize the portion of the config that is actually the plug-in config? The gaping hole in this is the second part of step 2. What is the 'approved' way of resolving references that my assembly doesn't know about? – Interject

The way that we handle this (so it is approved by me :)) is to retrieve the list of assemblies and their required assemblies (for example, third party control libraries) and store them in the exe directory, then load the plugin using Assembly.LoadFile. – Snort

In addition, for remote deployments, we keep a manifest on the server that specifies the DLL and some key attributes (version and size, primarily). Then, when the remote app asks for an update, it sends the files/versions that it knows about and we send back a zipped file containing all of the DLLs that need to be updated. The client then unzips this file into the directory and loads the key plugins. Updating is a bit tricky since you have to deal with locked files, but if none of your plugin dlls are used by the main app, you should be good. – Snort

Also if it helps describe the situation, the plug-ins 'configuration' is implemented as a DynamicObject backed by a dictionary so really what is happening under the covers is I am serializing a dictionary with some additional metadata (such as assembly as Type names) into the configuration to get down to the lowest common denominator and then on 'rehydration' I deserialize the dictionary, resolve the type based on known types (in the primary appdomain) and then overwrite the dynamically created object's underlying dictionary with the one I've rehydrated. – Interject

'store them in the exe directory' - This is at least one of the problems. The plug-ins used in any specific configuration cannot be known until we deserialize the config. And if I have 10 different 'servers' (more realistically network workstations) who all could potentially run the UI (which as stated in the question also requires knowledge of known plugins) there is no way to distribute every possible plug-in in the exe directory. – Interject

I like your approach, I really do, but I've gotten past that hump already by including the assembly and type information in the serialized dynamic types already. It probably seems like I am being difficult but I guess even with the (obscenely) long question I've unintentionally excluded necessary detail to solve the problem. – Interject

I'm going to award you the bounty because of your comment describing your approach to updatable assemblies, though it doesn't quite solve my problem. Please see the update I posted today to better understand the context of the general problem that led to my question. I hope it provides the detail necessary to answer the question. Thank for all of your help! – Interject

I appreciate that. I haven't forgotten about you, I just had a big project due. I will contemplate this over the next couple of days and see what I can come up with. – Snort

Answer 3

Maybe you can divide this problem into two

1. administrator allow users to download one of predefined configuration (set of libraries) and MEF helps to inject required dependencies

2. each activity from user should pass through security proxy, plugin modules not allowed call BL directly. Proxy could match custom security attribute and allowed activities. i.e.

   [MyRole(Name = new[] { "Security.Action" })] void BlockAccount(string accountId){}

   [MyRole(Name = new[] { "Manager.Action" })] void CreateAccount(string userName){}

   [MyRole(Name = new[] { "Security.View", "Manager.View" })] List<> AcountList(Predicate p){}

and allow for AD groups (some abstract description)

1. corp\securityOperators = "Security.*" //allow calls to all security manipulation
2. corp\HQmanager = "Manager.View" //allow only view access
3. corp\Operator = "Manager.*"

Almandine

Perhaps I'm just dense, but I'm having a hard time understanding how this relates/answers the question. – Interject

Answer 4

I'm not sure I completely understand the problem but I think this situation calls for "type-preserving serialization" - that is, the serialized file contains enough type information to deserialize back to the original object graph without any hints from the calling application as to what types are involved.

I've used Json.NET to do this and I can highly recommend the library for type-preserving serialization of object graphs. It looks like the NetDataContractSerializer can also do this, from the MSDN Remarks

The NetDataContractSerializer differs from the DataContractSerializer in one important way: the NetDataContractSerializer includes CLR type information in the serialized XML, whereas the DataContractSerializer does not. Therefore, the NetDataContractSerializer can be used only if both the serializing and deserializing ends share the same CLR types.

I chose Json.NET because it can serialize POCOs without any special attributes or interfaces. Both Json.NET and the NetDataContractSerializer allow you to use a custom SerializationBinder - in here you could put any logic regarding loading assemblies that may not yet be loaded.

Unfortunately, changing serialization schemes might be the "breaking-est" change to suggest because all your existing files will become incompatible. You might be able to write a conversion utility that deserializes a file using the old method and serializes the resulting object graph using the new method.

Spell

I appreciate the suggestion, however determining which assemblies/types are referenced by the configuration is has already been addressed in my implementation. The question is more about the approach of discovering/transferring/accessing the referenced assemblies in a client/server environment. – Interject

I've added another update to my question which will hopefully help explain better what I'm after. – Interject

OK, I think your update narrows in on the problem (at least for my understanding) but there's still a lot of detail that is not directly relevant. Detailed context can be good because people can suggest a better approach, but the difference between the question and the context should be clear. As it's written now, I have a hard time distinguishing your question from your context. Anyway, good luck (I would probably just do the file share, BTW). – Spell