Out-of-proc COM server stuck

Question

I'm using out-of-proc COM server (COM singleton "Engine" implemented using DECLARE_CLASSFACTORY_SINGLETON), it works in STA (CComSingleThreadModel, _ATL_APARTMENT_THREADED).

COM server clients:

1. ActiveScript (JScript), (I pass Engine reference using AddNamedItem).
2. Two independent IE BHOs.

BHOs periodically call Engine::dispatchEvent, Engine calls JavaScript functions of ActiveScript. This architecture worked perfectly until I turned on two BHOs simultaneously.

If I turn on two BHOs, stuck occurs when I call function of ActiveScript (using IDispatch/Invoke). I don't create any additional threads.

Some notes:

• If I don't pass object retrieved from BHO to ActiveScript (or replace it with the same object created in Engine) everything works fine.
• Stuck occurs only when JScript garbage collector tries to release object retrieved from BHO (IUnknown_Release_Proxy in callstack).

Callstack:

>    ntdll.dll!_ZwWaitForMultipleObjects@20()  + 0x15 bytes    
 ntdll.dll!_ZwWaitForMultipleObjects@20()  + 0x15 bytes    
 KernelBase.dll!_WaitForMultipleObjectsEx@20()  + 0x100 bytes    
 kernel32.dll!_WaitForMultipleObjectsExImplementation@20()  + 0x8e bytes    
 user32.dll!_RealMsgWaitForMultipleObjectsEx@20()  + 0xe2 bytes    
 ole32.dll!CCliModalLoop::BlockFn(void * * ahEvent, unsigned long cEvents, unsigned long * lpdwSignaled)  Line 1222    C++
 ole32.dll!ModalLoop(CMessageCall * pcall)  Line 211    C++
 ole32.dll!ThreadSendReceive(CMessageCall * pCall)  Line 4979    C++
 ole32.dll!CRpcChannelBuffer::SwitchAptAndDispatchCall(CMessageCall * * ppCall)  Line 4454 + 0x6 bytes    C++
 ole32.dll!CRpcChannelBuffer::SendReceive2(tagRPCOLEMESSAGE * pMessage, unsigned long * pstatus)  Line 4076    C++
 ole32.dll!CCliModalLoop::SendReceive(tagRPCOLEMESSAGE * pMsg, unsigned long * pulStatus, IInternalChannelBuffer * pChnl)  Line 899 + 0x17 bytes    C++
 ole32.dll!CAptRpcChnl::SendReceive(tagRPCOLEMESSAGE * pMsg, unsigned long * pulStatus)  Line 583 + 0xd bytes    C++
 ole32.dll!CCtxComChnl::SendReceive(tagRPCOLEMESSAGE * pMessage, unsigned long * pulStatus)  Line 734 + 0xa bytes    C++
 ole32.dll!NdrExtpProxySendReceive(void * pThis, _MIDL_STUB_MESSAGE * pStubMsg)  Line 1932    C++
 rpcrt4.dll!@NdrpProxySendReceive@4()  + 0xe bytes    
 rpcrt4.dll!_NdrClientCall2()  + 0x144 bytes    
 ole32.dll!ObjectStublessClient(void * ParamAddress, long Method)  Line 474 + 0x8 bytes    C++
 ole32.dll!_ObjectStubless@0()  Line 154    Asm
 ole32.dll!RemoteReleaseRifRefHelper(IRemUnknown * pRemUnk, int fReleaseRemUnkProxy, int fProcessingPostedMessage, OXIDEntry * pOXIDEntry, unsigned short cRifRef, tagREMINTERFACEREF * pRifRef, IUnknown * pAsyncRelease)  Line 6770 + 0xc bytes    C++
 ole32.dll!RemoteReleaseRifRef(CStdMarshal * pMarshal, OXIDEntry * pOXIDEntry, unsigned short cRifRef, tagREMINTERFACEREF * pRifRef)  Line 6694    C++
 ole32.dll!CStdMarshal::DisconnectCliIPIDs()  Line 3964    C++
 ole32.dll!CStdMarshal::Disconnect(unsigned long dwType)  Line 3273    C++
 ole32.dll!CStdIdentity::~CStdIdentity()  Line 312    C++
 ole32.dll!CStdIdentity::`scalar deleting destructor'()  + 0xd bytes    C++
 ole32.dll!CStdIdentity::CInternalUnk::Release()  Line 767    C++
 ole32.dll!IUnknown_Release_Proxy(IUnknown * This)  Line 1773    C++
 oleaut32.dll!_VariantClear@4()  + 0xac9 bytes    
 jscript.dll!VAR::Clear()  + 0x50 bytes    
 jscript.dll!GcAlloc::ReclaimGarbage()  + 0xa2 bytes    
 jscript.dll!GcContext::Reclaim()  + 0x8e bytes    
 jscript.dll!GcContext::CollectCore()  - 0x72f bytes    
 jscript.dll!GcContext::Collect()  + 0x34 bytes    
 jscript.dll!CScriptRuntime::Run()  - 0x864f bytes    
 jscript.dll!ScrFncObj::CallWithFrameOnStack()  + 0xf3 bytes    
 jscript.dll!ScrFncObj::Call()  + 0x84 bytes    
 jscript.dll!NameTbl::InvokeInternal()  + 0x113 bytes    
 jscript.dll!VAR::InvokeByDispID()  + 0x73 bytes    
 jscript.dll!CScriptRuntime::Run()  + 0x1d89 bytes    
 jscript.dll!ScrFncObj::CallWithFrameOnStack()  + 0xf3 bytes    
 jscript.dll!ScrFncObj::Call()  + 0x84 bytes    
 jscript.dll!NameTbl::InvokeInternal()  + 0x113 bytes    
 jscript.dll!VAR::InvokeByDispID()  + 0x73 bytes    
 jscript.dll!CScriptRuntime::Run()  + 0x1d89 bytes    
 jscript.dll!ScrFncObj::CallWithFrameOnStack()  + 0xf3 bytes    
 jscript.dll!ScrFncObj::Call()  + 0x84 bytes    
 jscript.dll!NameTbl::InvokeInternal()  + 0x12c6 bytes    
 jscript.dll!VAR::InvokeByDispID()  + 0x73 bytes    
 jscript.dll!NameTbl::GetVal()  + 0x3b bytes

Implementation details:

// Engine (out of process COM singleton)

class ATL_NO_VTABLE CEngine :
    public CComObjectRootEx<CComSingleThreadModel>,
    public CComCoClass<CEngine, &CLSID_Engine>,
    public IDispatchImpl<IEngine, &IID_IEngine, &LIBID_EngineLib, /*wMajor =*/ 1, /*wMinor =*/ 0>
{

    DECLARE_CLASSFACTORY_SINGLETON(CEngine)

    STDMETHOD(dispatchEvent)(BSTR name, IDispatch* pEvent, VARIANT_BOOL* pbSuccess)
    {
        // pEvent is CPropertyStore instance
        ActiveScriptDispatch.Invoke1(L"FuncName", pEvent, &varResult);
    }
}


// BHO

class CPropertyStore :
    public CComObjectRootEx<CComSingleThreadModel>,
    public CComCoClass<CPropertyStore, &CLSID_NULL>,
    public IDispatch
{
    BEGIN_COM_MAP(CPropertyStore)
        COM_INTERFACE_ENTRY(IUnknown)
        COM_INTERFACE_ENTRY(IDispatch)
    END_COM_MAP()

    BOOL SetProperty(CString strName, VARIANT *value)
    {
        // Store value in CAtlArray
    }

    // IDispatch impl
    STDMETHOD(GetTypeInfoCount)(UINT *pctinfo);
    STDMETHOD(GetTypeInfo)(UINT iTInfo, LCID lcid, ITypeInfo **ppTInfo);
    STDMETHOD(GetIDsOfNames)(REFIID riid, LPOLESTR *rgszNames, UINT cNames, LCID lcid, DISPID *rgDispId);
    STDMETHOD(Invoke)(DISPID dispIdMember, REFIID riid, LCID lcid, WORD wFlags, DISPPARAMS *pDispParams, 
        VARIANT *pVarResult,EXCEPINFO *pExcepInfo, UINT *puArgErr);
}

class ATL_NO_VTABLE CBHO :
    public CComObjectRootEx<CComSingleThreadModel>,
    public CComCoClass<CBHO, &CLSID_BHO>,
    public IObjectWithSiteImpl<CBHO>,
    public IDispatchImpl<IBHO, &IID_IBHO, &LIBID_Lib, /*wMajor =*/ 1, /*wMinor =*/ 0>,
    public IDispEventImpl<1, CBHO, &DIID_DWebBrowserEvents2, &LIBID_SHDocVw, 1, 0>
{
    void onEvent(...)
    {
        if(m_pEngine == NULL && SUCCEEDED(m_pEngine.CoCreateInstance(CLSID_Engine)))
        {
            CComObject<CPropertyStore> *pEvent = NULL;
            HRESULT hRes = CComObject<CPropertyStore>::CreateInstance(&pEvent);

            CComVariant varEvent(pEvent);
            CComVariant varName(L"EventName");
            CComVariant varResult;

            m_pEngine.Invoke2(L"dispatchEvent", &varName, &varEvent, &varResult);
        }
    }
}

Nonconcurrence

Yes, deadlock due to the required thread context switch when Javascript tries to release your object. You need to look at the thread that originally created the object and see why it is not being responsive. If it is blocked then you need MsgWaitForMultipleObjectsEx to permit this marshaled call to complete. – Centeno

BHO thread is not responding because it's waiting for dispatchEvent result (oleaut32.dll!_IDispatch_Invoke_Proxy, user32.dll!_RealMsgWaitForMultipleObjectsEx in callstack). – Nonconcurrence

I have added implementation details. – Nonconcurrence

Looks like scripting engine (it's garbage collector) is releasing an object that belongs to different apartment. Your engine is STA, is scripting operation in MTA, or in another STA? If it is MTA, you should redesign to not block STA for the time of the call (e.g. dispatchEvent from a worker thread, not sure if this is possible without seeing code). – Simpatico

Scripting operation should be in STA, ActiveScript created in Engine using CoCreateInstance(L"JScript"). – Nonconcurrence

@Nonconcurrence is there any update on this? I have a similar problem. – Rancor

@Rancor i haven't solved the issue, but found a workaround - if i pass string (BSTR) instead of object (IDispatch) everything works fine. – Nonconcurrence

@KAdot, did you try to debug into your object's Release, e.g. with a breakpoint? – Saleable

Answer 1

Your BHO (Browser Helper Object) is in a single thread appartment. Every COM call in a STA that is made to an object on another STA (different thread) is ordered by a message in a message queue before it is "transformed" in a method call.

This is usually not a problem, because most of the time calls are triggered by the GUI who is single threaded. COM calls wait their turn along with WM_LBUTTONUP messages and such.

What happens in your case is that while servicing onEvent, you send your BHO object to another thread, in another process, your out-of-process COM object. When you try to call back to the original object from your MTA appartment, a Windows message is posted to your BHO STA thread in the Internet Explorer process that hosts it. But the message queue is still busy servicing the original request.

That explains your deadlock, and why passing a string works.

Minda

This isn't entirely correct. Only cross-apartment calls to an STA object start by message queue processing. Also, regular STA (versus an STA with a custom IMessageFilter or WInRT's ASTA) processes messages when cross-apartment calls are made from it, so it is reentrant and it shouldn't deadlock like that. I think the op didn't provide enough details to solve the problem, the deadlock is elsewhere. – Saleable

You are right saying that only cross apartment calls use Windows messages, I will update my answer. But I remember having "stumbled" upon such a dead lock and non-reentrancy of Windows messages was the problem. I admit it was over 10 years ago. This post says the same thing as you, my memory must be failing me... – Minda