DirectX Partial Screen Capture

Question

I am trying to create a program that will capture a full screen directx application, look for a specific set of pixels on the screen and if it finds it then draw an image on the screen.

I have been able to set up the application to capture the screen the directx libraries using the code the answer for this question Capture screen using DirectX

In this example the code saves to the harddrive using the IWIC libraries. I would rather manipulate the pixels instead of saving it.

After I have captured the screen and have a LPBYTE of the entire screen pixels I am unsure how to crop it to the region I want and then being able to manipulate the pixel array. Is it just a multi dimensional byte array?

The way I think I should do it is

1. Capture screen to IWIC bitmap (done).
2. Convert IWIC bitmap to ID2D1 bitmap using ID2D1RenderTarget::CreateBitmapFromWicBitmap
3. Create new ID2D1::Bitmap to store partial image.
4. Copy region of the ID2D1 bitmap to a new bitmap using ID2D1::CopyFromBitmap.
5. Render back onto screen using ID2D1 .

Any help on any of this would be so much appreciated.

Partitive

Not sure if this would help you or not; but it seems that it may be relevant; check out this section of tutorials from rastertek that works with rendering to textures rastertek.com/dx11tut22.html – Embower

You don't have to get the whole screen in the first place, the original code can be adapted to capture only a rectangular portion of the screen. You'll get a buffer of pixels that you don't have to save if you don't want to. It's not clear what you want to do with these pixels. – Parrott

That was my first thought, I would much prefer to just capture part of the screen but I have no idea how. I thought it would be as easy as modifying the bounds (of the D3DPRESENT_PARAMETERS) of the capture but I cant seem to get it right. – Partitive

I have this code if you just need that. PS: add a @ with other's alias to notify them, I just happen to pass by – Parrott

@SimonMourier Hi yes please I would really appreciate that. – Partitive

Answer 1

Here is a modified version of the original code that only captures a portion of the screen into a buffer, and also gives back the stride. Then it browses all the pixels, dumps their colors as a sample usage of the returned buffer.

In this sample, the buffer is allocated by the function, so you must free it once you've used it:

// sample usage
int main()
{
  LONG left = 10;
  LONG top = 10;
  LONG width = 100;
  LONG height = 100;
  LPBYTE buffer;
  UINT stride;
  RECT rc = { left, top, left + width, top + height };
  Direct3D9TakeScreenshot(D3DADAPTER_DEFAULT, &buffer, &stride, &rc);

  // In 32bppPBGRA format, each pixel is represented by 4 bytes
  // with one byte each for blue, green, red, and the alpha channel, in that order.
  // But don't forget this is all modulo endianness ...
  // So, on Intel architecture, if we read a pixel from memory
  // as a DWORD, it's reversed (ARGB). The macros below handle that.

  // browse every pixel by line
  for (int h = 0; h < height; h++)
  {
    LPDWORD pixels = (LPDWORD)(buffer + h * stride);
    for (int w = 0; w < width; w++)
    {
      DWORD pixel = pixels[w];
      wprintf(L"#%02X#%02X#%02X#%02X\n", GetBGRAPixelAlpha(pixel), GetBGRAPixelRed(pixel), GetBGRAPixelGreen(pixel), GetBGRAPixelBlue(pixel));
    }
  }

  // get pixel at 50, 50 in the buffer, as #ARGB
  DWORD pixel = GetBGRAPixel(buffer, stride, 50, 50);
  wprintf(L"#%02X#%02X#%02X#%02X\n", GetBGRAPixelAlpha(pixel), GetBGRAPixelRed(pixel), GetBGRAPixelGreen(pixel), GetBGRAPixelBlue(pixel));

  SavePixelsToFile32bppPBGRA(width, height, stride, buffer, L"test.png", GUID_ContainerFormatPng);
  LocalFree(buffer);
  return 0;;
}

#define GetBGRAPixelBlue(p)         (LOBYTE(p))
#define GetBGRAPixelGreen(p)        (HIBYTE(p))
#define GetBGRAPixelRed(p)          (LOBYTE(HIWORD(p)))
#define GetBGRAPixelAlpha(p)        (HIBYTE(HIWORD(p)))
#define GetBGRAPixel(b,s,x,y)       (((LPDWORD)(((LPBYTE)b) + y * s))[x])

int main()

HRESULT Direct3D9TakeScreenshot(UINT adapter, LPBYTE *pBuffer, UINT *pStride, const RECT *pInputRc = nullptr)
{
  if (!pBuffer || !pStride) return E_INVALIDARG;

  HRESULT hr = S_OK;
  IDirect3D9 *d3d = nullptr;
  IDirect3DDevice9 *device = nullptr;
  IDirect3DSurface9 *surface = nullptr;
  D3DPRESENT_PARAMETERS parameters = { 0 };
  D3DDISPLAYMODE mode;
  D3DLOCKED_RECT rc;

  *pBuffer = NULL;
  *pStride = 0;

  // init D3D and get screen size
  d3d = Direct3DCreate9(D3D_SDK_VERSION);
  HRCHECK(d3d->GetAdapterDisplayMode(adapter, &mode));

  LONG width = pInputRc ? (pInputRc->right - pInputRc->left) : mode.Width;
  LONG height = pInputRc ? (pInputRc->bottom - pInputRc->top) : mode.Height;

  parameters.Windowed = TRUE;
  parameters.BackBufferCount = 1;
  parameters.BackBufferHeight = height;
  parameters.BackBufferWidth = width;
  parameters.SwapEffect = D3DSWAPEFFECT_DISCARD;
  parameters.hDeviceWindow = NULL;

  // create device & capture surface (note it needs desktop size, not our capture size)
  HRCHECK(d3d->CreateDevice(adapter, D3DDEVTYPE_HAL, NULL, D3DCREATE_SOFTWARE_VERTEXPROCESSING, &parameters, &device));
  HRCHECK(device->CreateOffscreenPlainSurface(mode.Width, mode.Height, D3DFMT_A8R8G8B8, D3DPOOL_SYSTEMMEM, &surface, nullptr));

  // get pitch/stride to compute the required buffer size
  HRCHECK(surface->LockRect(&rc, pInputRc, 0));
  *pStride = rc.Pitch;
  HRCHECK(surface->UnlockRect());

  // allocate buffer
  *pBuffer = (LPBYTE)LocalAlloc(0, *pStride * height);
  if (!*pBuffer)
  {
    hr = E_OUTOFMEMORY;
    goto cleanup;
  }

  // get the data
  HRCHECK(device->GetFrontBufferData(0, surface));

  // copy it into our buffer
  HRCHECK(surface->LockRect(&rc, pInputRc, 0));
  CopyMemory(*pBuffer, rc.pBits, rc.Pitch * height);
  HRCHECK(surface->UnlockRect());

cleanup:
  if (FAILED(hr))
  {
    if (*pBuffer)
    {
      LocalFree(*pBuffer);
      *pBuffer = NULL;
    }
    *pStride = 0;
  }
  RELEASE(surface);
  RELEASE(device);
  RELEASE(d3d);
  return hr;
}

Parrott

That is amazing, thanks so much @SimonMourier. I have just one last question if that is ok? I want to now access the pixel array (as a two dimensional array i am guessing) so I can look for pixels of a specific color. Would I just use the pixels LPBYTE pointer as the start of the pixel array and use the stride as the size of each pixel after given the height and width? Do you have any code to example this by any chance? – Partitive

Fantastic. Thanks so much @SimonMourier – Partitive