Callback from .Net COM dll to Delphi client in registration-free (side-by-side) COM

Question

TLDR: I'm trying to call async callbacks from a .Net COM dll to Delphi client .exe, but these does not seem to work properly in registration-free COM, while synchronous callbacks do work, and also async callbacks are working when it's not a reg-free COM.

---

My global case is that I'm having a foreign closed-source .Net dll that exposes some public events. I need to pass these events to Delphi app. So I decided to make an intermediate .dll that would work as a COM bridge between my app and that another dll. It worked just fine when my dll is registered via regasm, but things are getting worse when I switch to reg-free COM. I shortened my case to small reproducible example which does not depend on the other dll, so I'll be posting it below.

Based on this answer I made a public interface ICallbackHandler which I expect to get from Delphi client app:

namespace ComDllNet
{
    [ComVisible(true)]
    [Guid("B6597243-2CC4-475B-BF78-427BEFE77346")]
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    public interface ICallbackHandler
    {
        void Callback(int value);
    }

    [ComVisible(true)]
    [Guid("E218BA19-C11A-4303-9788-5A124EAAB750")]
    public interface IComServer
    {
        void SetHandler(ICallbackHandler handler);
        void SyncCall();
        void AsyncCall();
    }

    [ComVisible(true)]
    [Guid("F25C66E7-E9EF-4214-90A6-3653304606D2")]
    [ClassInterface(ClassInterfaceType.None)]
    public sealed class ComServer : IComServer
    {
        private ICallbackHandler handler;
        public void SetHandler(ICallbackHandler handler) { this.handler = handler; }

        private int GetThreadInfo()
        {
            return Thread.CurrentThread.ManagedThreadId;
        }

        public void SyncCall()
        {
            this.handler.Callback(GetThreadInfo());
        }

        public void AsyncCall()
        {
            this.handler.Callback(GetThreadInfo());
            Task.Run(() => {
                for (int i = 0; i < 5; ++i)
                {
                    Thread.Sleep(500);
                    this.handler.Callback(GetThreadInfo());
                }
            });
        }
    }
}

Then, I gave a strong name to dll, and registered it via Regasm.exe.

Now I turned to Delphi client. I create the tlb wrapper code using Component > Import Component > Import a Type Library which gave me

  ICallbackHandler = interface(IUnknown)
    ['{B6597243-2CC4-475B-BF78-427BEFE77346}']
    function Callback(value: Integer): HResult; stdcall;
  end;
  IComServer = interface(IDispatch)
    ['{E218BA19-C11A-4303-9788-5A124EAAB750}']
    procedure SetHandler(const handler: ICallbackHandler); safecall;
    procedure SyncCall; safecall;
    procedure AsyncCall; safecall;
  end;
  IComServerDisp = dispinterface
    ['{E218BA19-C11A-4303-9788-5A124EAAB750}']
    procedure SetHandler(const handler: ICallbackHandler); dispid 1610743808;
    procedure SyncCall; dispid 1610743809;
    procedure AsyncCall; dispid 1610743810;
  end;

And created a handler and some Form with two buttons and memo to test things:

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, ComDllNet_TLB, StdCtrls;

type
  THandler = class(TObject, IUnknown, ICallbackHandler)
  private
    FRefCount: Integer;
  protected
   function Callback(value: Integer): HResult; stdcall;

   function QueryInterface(const IID: TGUID; out Obj): HRESULT; stdcall;
   function _AddRef: Integer; stdcall;
   function _Release: Integer; stdcall;
  public
    property RefCount: Integer read FRefCount;
  end;

type
  TForm1 = class(TForm)
    Memo1: TMemo;
    syncButton: TButton;
    asyncButton: TButton;
    procedure FormCreate(Sender: TObject);
    procedure syncButtonClick(Sender: TObject);
    procedure asyncButtonClick(Sender: TObject);
  private
    { Private declarations }
    handler : THandler;
    server : IComServer;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

function THandler._AddRef: Integer;
begin
  Inc(FRefCount);
  Result := FRefCount;
end;

function THandler._Release: Integer;
begin
  Dec(FRefCount);
  if FRefCount = 0 then
  begin
    Destroy;
    Result := 0;
    Exit;
  end;
  Result := FRefCount;
end;

function THandler.QueryInterface(const IID: TGUID; out Obj): HRESULT;
const
  E_NOINTERFACE = HRESULT($80004002);
begin
  if GetInterface(IID, Obj) then
    Result := 0
  else
    Result := E_NOINTERFACE;
end;

function THandler.Callback(value: Integer): HRESULT;
 begin
  Form1.Memo1.Lines.Add(IntToStr(value));
  Result := 0;
 end;

procedure TForm1.FormCreate(Sender: TObject);
 begin
  handler := THandler.Create();
  server := CoComServer.Create();
  server.SetHandler(handler);
 end;

procedure TForm1.syncButtonClick(Sender: TObject);
 begin
  Form1.Memo1.Lines.Add('Begin sync call');
  server.SyncCall();
  Form1.Memo1.Lines.Add('End sync call');
 end;

procedure TForm1.asyncButtonClick(Sender: TObject);
 begin
  Form1.Memo1.Lines.Add('Begin async call');
  server.AsyncCall();
  Form1.Memo1.Lines.Add('End async call');
 end;

end.

So, I run it, pressed 'sync' and 'async' buttons and everything worked as expected. Note how the thread ids of a Task comes after 'End async call' line (also with some delay because of Thread.Sleep):

End of part one. Now I switched to using Rregistration-free (side-by-side) COM. Based on this answer I added dependentAssembly part to my Delphi app manifest:

<dependency>
    <dependentAssembly>
        <assemblyIdentity name="ComDllNet" version="1.0.0.0" publicKeyToken="f31be709fd58b5ba" processorArchitecture="x86"/>
    </dependentAssembly>
</dependency>

Using the mt.exe tool I generated a manifest for my dll:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
    <assemblyIdentity name="ComDllNet" version="1.0.0.0" publicKeyToken="f31be709fd58b5ba" processorArchitecture="x86"/>
    <clrClass clsid="{F25C66E7-E9EF-4214-90A6-3653304606D2}" progid="ComDllNet.ComServer" threadingModel="Both" name="ComDllNet.ComServer" runtimeVersion="v4.0.30319"/>
    <file name="ComDllNet.dll" hashalg="SHA1"/>
</assembly>

Then I unregistered the dll and run the app. And I found that only synchronous parts of the callbacks are working:

Edit: Note that you have to unregister with /tlb option, otherwise it will continue working on local machine, as if dll was still registered (see).

I tired a number of things already, and I'm not sure what to do next. I'm staring to suspect that the initial approach should not work at all and I need to implement some threading on the Delphi app side. But I'm not sure what and how. Any help would be appreciated!

Tahitian

Answer 1

You have to register the ICallbackHandler interface. So, in the same file where you have the clrClass element, but as a sibling of the file elements, add:

    <comInterfaceExternalProxyStub iid="{B6597243-2CC4-475B-BF78-427BEFE77346}"
                                   name="ICallbackHandler"
                                   tlbid="{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}"
                                   proxyStubClsid32="{00020424-0000-0000-C000-000000000046}"/>

This tells COM to use an external proxy/stub, the type library marshaler ({00020424-0000-0000-C000-000000000046}), and it tells the type library marshaler to look for your type library ({XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}). This GUID is your assembly's GUID, found in your project's properties (check AssemblyInfo.cs).

You need to generate this type library. Since you want registration-free COM, I think TLBEXP.EXE fits the bill perfectly, you can set it up as a post build event.

Finally, you can keep a separate type library file or you can embed it in your assembly. I advise you keep it separate, even more so if your assembly is big.

Either way, you need to put this into the manifest. Here's an example using a separate .TLB file:

    <file name="ComDllNet.tlb">
        <typelib tlbid="{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}"
                 version="1.0"
                 helpdir="."
                 flags=""/>
    </file>

If you embed the type library, add the following as a child of the <file name="ComDLLNet.dll"/> element:

        <typelib tlbid="{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}"
                 version="1.0"
                 helpdir="."
                 flags=""/>

Landowner

Registering the proxy is the right thing, but @Tahitian also shouldn't be calling this.handler.Callback(GetThreadInfo()) directly from a pool thread, without COM marshaling. Unless his Delphi side is expecting to receive the callback on a random thread. – Sherronsherry

I tried adding comInterfaceExternalProxyStub element but referencing to .dll element itself. Now I tried your proposal with sepearate file element for .tlb and it haven't changed anything, but seem more adequate, so I'll be looking to keep it. – Tahitian

@Noseratio : this seem to be indeed the case. Could you please guide me on proper marshalling implementation? – Tahitian

@Mikhail, the easiest IMO two methods are CoMarshalInterThreadInterfaceInStream/CoGetInterfaceAndReleaseStream (example here) and Global Interface Table (GIT, example here). – Sherronsherry

On the other hand, things are working fine when the .dll is registrered. And it also works fine if I register dll with /tlb key and then unregister it without /tlb key. So it seems that it's indeed more on explaining Sxs about things in my assembly/tlb. Still I'm not sure what I'm doing wrong now. I am to add the typelib and comInterfaceExternalProxyStub elements into the .dll manifest, not into .exe manifest, right? – Tahitian

@Mikhail, yes, add this information to the manifest that refers to your DLL. If you embed the manifest in the DLL, then it's in the DLL's manifest. – Landowner

Finally it worked with original explanation from the answer! Still it did not work when I tired it first time, and it was because of some kind of caching. I was able to totally corrupt the .dll manifest file and everything were running as if it was OK. The system noticed that manifest changed only after rebuilding .exe, and not always on first. Pretty strange. Anyway, thanks for the help! – Tahitian

@Noseratio, unless the Delphi object aggregates the free-threaded marshaler (or implements IMarshal in a similar fashion), there's no problem in .NET invoking an RCW. – Landowner

Paulo, from that link: "If you create an RCW in one application domain or apartment, and then pass a reference to another application domain or apartment, a proxy to the first object will be used." I.e., RCW doesn't do any COM marshaling. This is exactly the problem I want @Tahitian to be aware of. My comment is too long, posting it as an answer. – Sherronsherry

Answer 2

This is too long to be a comment, so posting it as an answer.

A pointer to COM interfaces should never be accessed from a different COM apartment without proper marshaling. In this case, this.handler is (most likely) an STA COM object created on an Delphi's STA thread. Then it directly gets called from a .NET MTA pool thread thread inside Task.Run, without any kind of COM marshaling. This is a violation of COM hard rules, outlined here INFO: Descriptions and Workings of OLE Threading Models.

The same is true about a managed RCW proxy wrapping a COM interface on the .NET side. The RCW will just marshal the method call from managed to unmanaged code, but it won't do anything about COM marshaling.

This can lead to all kind of nasty surprised, especially if the OP accesses the Delphi app's UI inside handler.Callback.

Now, it's possible that the handler object aggregates the Free Threaded Marshaler (this would have its own rules to follow, and I doubt it's the case with the OP code). Let it be so, the pointer to handler object will indeed be unmarshaled to the same pointer by the FTM. However, the server code which calls the object from another thread (i.e., Task.Run(() => { ... this.handler.Callback(GetThreadInfo() ...}) should never assume the COM object is free-threaded, and it still should do the correct marshaling. If lucky, the direct pointer will be given back when unmarshaling.

There's a bunch of methods to do the marshaling:

• CoMarshalInterThreadInterfaceInStream/CoGetInterfaceAndReleaseStream.
• CoMarshalInterface/CoUnmarshalInterface.
• Global Interface Table (GIT).
• CreateObjrefMoniker/BindMoniker.
• etc.

Of course, for the above marshaling methods to work, the correct COM proxy/stub classes should be registered or provisioned via a side-by-side manifest, as Paulo Madeira's answer explains.

Alternatively, a custom dispinterface can be used (in which case all calls would go through IDispatch with OLE Automation marshaler), or any other standard COM interface known to the standard COM marshaler. I often use IOleCommandTarget for simple callbacks, it doesn't require anything to be registered.

Sherronsherry

Have you actually tried to acquire an RCW (to the actual object or a proxy/stub) in one apartment and using it in another one? I've never worried about that, and this blog post is quite explicit about what amount of backstage work an RCW does. – Landowner

@PauloMadeira, I have, many times. Calling a method on an the same RCW across apartments is as good as calling a method on the underlying raw COM interface pointer across apartments. I.e., RCW is transparent in that respect, and the rest really depends on the particular COM object. You may be OK if you know for sure the object is correctly implemented as free-threaded (which I'm sure isn't the case about handler here). – Sherronsherry