Create Arbitrary Visual Studio COM Object (such as IDebugEngine2) from GUID

Question

I am investigating developing a new project system for a language not currently supported by Visual Studio.

There already exists a third party Debugger Adapter Protocol server for this language, however as it doesn't seem to work properly with the native DAP client used in Visual Studio I'd like to write my own AD7Engine and simply defer all the calls I don't need to modify back to the original implementation in Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.dll

In theory I feel this should be relatively simple; given the target COM object

namespace Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.AD7.Implementation
{
  [ComVisible(true)]
  [Guid("8355452D-6D2F-41b0-89B8-BB2AA2529E94")]
  internal class AD7Engine

I think I should be able to do something like

var type = Type.GetTypeFromCLSID(new Guid("8355452D-6D2F-41b0-89B8-BB2AA2529E94"));
var instance = Activator.CreateInstance(type);

However this fails, saying the class is not registered. Even if I try and create an instance of my own AD7Engine using this technique it says class not registered. The crux of this issue, I feel, is that the COM object is defined in the registry under HKCU\Software\Microsoft\VisualStudio\<version>\CLSID (or maybe under Visual Studio's private registry hive in newer versions), however I strongly suspect Activator.CreateInstance does not know about that location so can't see how to create it.

I've been pulling apart vsdebug.dll in IDA to see where the engine creation details come from; ultimately, it uses a very similar technique to this, this and this

v8 = GetLoader(a1, &rclsid);

if (v8 < 0)
    goto LABEL_26;
v8 = GetModulePath(a1, &bstrString);
if (v8 < 0)
{
    v4 = bstrString;
    goto LABEL_26;
}
v9 = CoCreateInstance(&rclsid, 0, dwClsContext, &_GUID_10e4254c_6d73_4c38_b011_e0049b2e0a0f, &ppv);
v4 = bstrString;
if (v9 < 0)
{
    v8 = -2147155456;
    goto LABEL_26;
}
v8 = (*(int(__stdcall * *)(LPVOID, BSTR, IID *, LPUNKNOWN, DWORD, LPVOID *))(*(_DWORD*)ppv + 28))(
    ppv,
    bstrString,
    a1,
    pUnkOuter,
    dwClsContext,
    v17);
if (v8 < 0)
{
    LABEL_26:
    v11 = CoCreateInstance(a1, pUnkOuter, dwClsContext, &IID_IUnknown, v17);
    if (v11 >= 0 || v8 != -2147155456)

however ultimately I feel I shouldn't have to resort to trying to calculate the VS CLSID registry path (which vsdebug.dll gets from somewhere unknown and everyone else just calculates or hardcodes); I feel there should be some high level way of creating arbitrary COM objects in the Visual Studio context, similar to when you use ServiceProvider.GetService().

For my purposes, this whole thing is a fun learning exercise, so whether this is a good idea or not is irrelevant to me; I'm committed to finding the answer at this stage; I just have to know what APIs exist for either

• achieving this outright with a simple API like GetService
• creating new instances of objects defined under the current Visual Studio version's CLSID registry key, or
• accessing keys under privateregistry.bin within a running extension

Any assistance would be greatly appreciated

Kelvin

Any chance the source code to your debug engine is available somewhere? – Euphony

After solving this question it became apparent to me that this technique wasn't as useful as I'd originally thought, as the VSCodeDebuggerHost debugger will call back to itself rather than my outer debugger whenever you try and do things like create breakpoints etc. As such, I don't currently have a working debug engine at this stage that can be shared – Kelvin

Answer 1

After doing a bit more research, the answer to all of these questions...is yes!

Given that Microsoft states that all of private application hives loaded from RegLoadAppKey must use the original handle obtained when opening the hive, I was very curious as to how Visual Studio was achieving this, considering I couldn't see any references to such a handle anywhere, yet when I stepped over registry functions in WinDbg, Process Monitor reported that the private hive had been accessed.

After stepping into these functions I discovered that actually Visual Studio detours all of the Win32 registry functions to its own special handler that determines whether or not it needs to redirect the function call.

As such, we can conclude

• Any attempts to access Visual Studio registry keys will automatically be redirected to Visual Studio's private application hive, if required

This still leaves us with the problem of having to construct the root Visual Studio key in the first place. As it turns out, you can use the VSRegistry.RegistryRoot method to gain a reference to various configuration stores (although only user and configuration types are actually supported)

And so, we can get a reference to the user's configuration key via

VSRegistry.RegistryRoot(__VsLocalRegistryType.RegType_Configuration);

This still leaves us with the problem of actually constructing the target COM object. As it happens, Visual Studio does have an API for this: ILocalRegistry3!

If you already know the CLSID of the object you wish to create, you can create it straight up with the CreateInstance method. In my case, somehow the GuidAttribute on the AD7Engine decompiled from Microsoft.VisualStudio.Debugger.VSCodeDebuggerHost.dll did not actually match the CLSID that was defined under the AD7Metrics key in the registry. As such, I consider it safer to translate the Engine ID GUID to the CLSID of the AD7Engine COM object, and then create the instance.

Utilizing this technique it also makes it easy to create instances of the debuggers specified in the Microsoft.VisualStudio.ProjectSystem.Debug.DebugEngines class.

private IDebugEngine2 VsCreateDebugEngine(Guid engineId)
{
    var config = VSRegistry.RegistryRoot(__VsLocalRegistryType.RegType_Configuration);

    var subKeyPath = $"AD7Metrics\\Engine\\{{{engineId}}}";

    var engineKey = config.OpenSubKey(subKeyPath);

    if (engineKey == null)
        throw new ArgumentException($"Could not find an AD7Engine for GUID '{engineId}'");

    var clsid = engineKey.GetValue("CLSID")?.ToString();

    if (clsid == null)
        throw new InvalidOperationException($"GUID '{engineId}' does not have a CLSID value");

    var obj = VsCreateComObject(new Guid(clsid));

    return (IDebugEngine2) obj;
}

private object VsCreateComObject(Guid guid)
{
    var localRegistry = (ILocalRegistry3)Microsoft.VisualStudio.Shell.ServiceProvider.GlobalProvider.GetService(typeof(SLocalRegistry));

    Guid riid = VSConstants.IID_IUnknown;

    IntPtr ptr;

    var result = localRegistry.CreateInstance(
        guid,
        null,
        ref riid,
        (uint) Microsoft.VisualStudio.OLE.Interop.CLSCTX.CLSCTX_INPROC_SERVER,
        out ptr
    );

    if (result != VSConstants.S_OK)
        throw new ArgumentException($"Failed to retrieve GUID '{guid}', GUID may not exist");

    var obj = Marshal.GetObjectForIUnknown(ptr);

    return obj;
}

You can easily enumerate all available engines with the following helper

[DebuggerDisplay("Name = {Name}, EngineID = {EngineID}, {CLSID} = {CLSID}")]
class AD7Info
{
    public string Name { get; set; }

    public Guid EngineID { get; set; }

    public Guid? CLSID { get; set; }
}

private List<AD7Info> GetAD7Infos()
{
    var config = VSRegistry.RegistryRoot(__VsLocalRegistryType.RegType_Configuration);

    var engineKey = config.OpenSubKey("AD7Metrics\\Engine");

    return engineKey.GetSubKeyNames().Select(n =>
    {
        var entryKey = engineKey.OpenSubKey(n);

        var clsid = entryKey.GetValue("CLSID")?.ToString();

        return new AD7Info
        {
            Name = entryKey.GetValue("Name")?.ToString(),
            EngineID = new Guid(n),
            CLSID = clsid != null ? new Guid(clsid) : (Guid?) null
        };
    }).ToList();
}

Example:

//Retrieve the "managed only" debugging engine
var result = VsCreateDebugEngine(DebuggerEngines.ManagedOnlyEngine);

or in my case

//Retrieve the VSCodeDebuggerHost debug engine
var result = VsCreateDebugEngine(new Guid("2833D225-C477-4388-9353-544D168F6030"));

I tested this on all 38 engines installed on my machine; most of them succeeded, potentially the ones that failed need to be created via their ProgramProvider or something.

Yay!

Kelvin