How to use glImportMemoryWin32HandleEXT to share an ID3D11Texture2D KeyedMutex Shared handle with OpenGL?

Question

I am investigating how to do cross-process interop with OpenGL and Direct3D 11 using the EXT_external_objects, EXT_external_objects_win32 and EXT_win32_keyed_mutex OpenGL extensions. My goal is to share a B8G8R8A8_UNORM texture (an external library expects BGRA and I can not change it. What's relevant here is the byte depth of 4) with 1 Mip-level allocated and written to offscreen with D3D11 by one application, and render it with OpenGL in another. Because the texture is being drawn to off-process, I can not use WGL_NV_DX_interop2.

My actual code can be seen here and is written in C# with Silk.NET. For illustration's purpose though, I will describe my problem in psuedo-C(++).

First I create my texture in Process A with D3D11, and obtain a shared handle to it, and send it over to process B.

#define WIDTH 100
#define HEIGHT 100
#define BPP 4 // BGRA8 is 4 bytes per pixel


ID3D11Texture2D *texture;

D3D11_TEXTURE2D_DESC texDesc = {
  .Width = WIDTH,
  .Height = HEIGHT,
  .MipLevels = 1,
  .ArraySize = 1,
  .Format = DXGI_FORMAT_B8G8R8A8_UNORM,
  .SampleDesc = { .Count = 1, .Quality = 0 }
  .Usage = USAGE_DEFAULT,
  .BindFlags = BIND_SHADER_RESOURCE
  .CPUAccessFlags = 0,
  .MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX
};

device->CreateTexture2D(&texDesc, NULL, &texture);

HANDLE sharedHandle;
texture->CreateSharedHandle(NULL, DXGI_SHARED_RESOURCE_READ, NULL, &sharedHandle);

SendToProcessB(sharedHandle, pid);

In Process B, I first duplicate the handle to get one that's process-local.

HANDLE localSharedHandle;
HANDLE hProcA = OpenProcess(PROCESS_DUP_HANDLE, false, processAPID);
DuplicateHandle(hProcA, sharedHandle, GetCurrentProcess(), &localSharedHandle, 0, false, DUPLICATE_SAME_ACCESS);
CloseHandle(hProcA)

At this point, I have a valid shared handle to the DXGI resource in localSharedHandle. I have a D3D11 implementation of ProcessB that is able to successfully render the shared texture after opening with OpenSharedResource1. My issue here is OpenGL however.

This is what I am currently doing for OpenGL

GLuint sharedTexture, memObj;
glCreateTextures(GL_TEXTURE_2D, 1, &sharedTexture);
glTextureParameteri(sharedTexture, TEXTURE_TILING_EXT, OPTIMAL_TILING_EXT); // D3D11 side is D3D11_TEXTURE_LAYOUT_UNDEFINED
// Create the memory object handle
glCreateMemoryObjectsEXT(1, &memObj);

// I am not actually sure what the size parameter here is referring to.
// Since the source texture is DX11, there's no way to get the allocation size, 
// I make a guess of W * H * BPP

// According to docs for VkExternalMemoryHandleTypeFlagBitsNV, NtHandle Shared Resources use HANDLE_TYPE_D3D11_IMAGE_EXT
glImportMemoryWin32HandleEXT(memObj, WIDTH * HEIGHT * BPP, GL_HANDLE_TYPE_D3D11_IMAGE_EXT, (void*)localSharedHandle);
DBG_GL_CHECK_ERROR(); // GL_NO_ERROR

Checking for errors along the way seems to indicate the import was successful. However I am not able to bind the texture.

if (glAcquireKeyedMutexWin32EXT(memObj, 0, (UINT)-1) {
  DBG_GL_CHECK_ERROR(); // GL_NO_ERROR
  glTextureStorageMem2D(sharedTexture, 1, GL_RGBA8, WIDTH, HEIGHT, memObj, 0); 
  DBG_GL_CHECK_ERROR(); // GL_INVALID_VALUE
  glReleaseKeyedMutexWin32EXT(memObj, 0);
}

What goes wrong is the call to glTextureStorageMem2D. The shared KeyedMutex is being properly acquired and released. The extension documentation is unclear as to how I'm supposed to properly bind this texture and draw it.

Inhuman

From the Vulkan documentation for the equivalent extension here khronos.org/registry/vulkan/specs/1.3-extensions/man/html/…, it seems that GL_HANDLE_TYPE_D3D11_IMAGE_EXT is indeed the correct flag to open the texture for an NtHandle shared texture. The issue now here is opening the mem object as a texture now. – Inhuman

Answer 1

After some more debugging, I managed to get [DebugSeverityHigh] DebugSourceApi: DebugTypeError, id: 1281: GL_INVALID_VALUE error generated. Memory object too small from the Debug context. By dividing my width in half I was able to get some garbled output on the screen.

It turns out the size needed to import the texture was not WIDTH * HEIGHT * BPP, (where BPP = 4 for BGRA in this case), but WIDTH * HEIGHT * BPP * 2. Importing the handle with size WIDTH * HEIGHT * BPP * 2 allows the texture to properly bind and render correctly.

Inhuman