First, the reason that VLC's content doesn't get captured when you use printscreen or other screen capture techniques is because (at least on older versions of Windows) they are using an 'overlay' to present the video. This is a special GPU construct that allows rendering to a virtual 'plane' above the normal screen surface. Since this bypasses everything else and goes straight to the GPU, there isn't any straightforward way to capture it. NOTE: In the last few releases of Windows the rules changed quite a bit concerning overlays. Apps can't assume that D3D9/DDraw overlays are supported and apps shouldn't use them nowadays since the system has much better methods of presenting content with the same high performance.
Direct3D 9 Overlays
If you are displaying video content using Direct3D 9, you too can use an overlay. See this page on MSDN for information on how to do it. There are a lot of restrictions on the usage of D3D9 overlays and they aren't supported on a lot of hardware, so I'll describe some other approaches.
This technique does not prevent other apps from injecting them into your address space and capturing your presents. Also, because it's not supported on some hardware and some capture APIs actually disable overlays, it doesn't provide very strong protection guarantees.
GPU-Based Content Protection
If you have a lot of time to learn about GPU content protection and you know you'll be displaying a non-standard DRM-protected video format, you can roll your own protected media path using GPU content protection. I'm not an expert in this area, and there are few who are. I wouldn't recommend this, but I wanted to point it out. This page on MSDN talks about how it's implemented in Direct3D 9 and this other page talks about how it's implemented using Direct3D 11.
This technique provides strong guarantees that the content has not been captured, since the key-exchange happens almost entirely through hardware (e.g. HDCP).
Media Foundation Protected Media Path (PMP)
If you are displaying video using a well-supported DRM-based media format, you can use Media Foundation's Protected Media Path, which makes use of the GPU-based content protection described previously. It also encapsulates most of the functionality in a separate protected process that other apps cannot intercept or otherwise interact with. If someone tries to install a test-signed driver or otherwise inject a binary that isn't code-signed by a trusted root authority, Windows will not allow the content to be decrypted and your content will remain secure. This technique provides strong guarantees that the content has not been captured. This is used by Netflix on Windows, Blueray players, and others.
DXGI Swap-chain Flags
Assuming you are presenting content using Direct3D 10.x/11.x (and hopefully now you are, as opposed to D3D 9, in 2014), you can use a number of flags on your swap-chain to lock down your content.
You can pass DXGI_SWAP_CHAIN_FLAG_RESTRICTED_CONTENT
in the swap-chain flags in order to fail the swap-chain creation if the system doesn't have HDCP or HDCP-like output protection. This probably isn't necessary for your purposes, but it's good if you are worrying about people capturing the HDMI or analog output.
The flag you definitely want is called DXGI_SWAP_CHAIN_FLAG_DISPLAY_ONLY
. This prevents all screen-capture APIs from ever seeing your swap-chain. It will just appear as a black rectangle to them.
See this page for all the DXGI swap-chain flags.
This technique doesn't provide the strong guarantees that GPU-based content protection provides, but if you are pretty confident that other apps aren't doing crazy things like injecting themselves into your address space and hooking your present calls, you can be be confident this prevents anyone else from seeing your content (except the monitor, of course).
Full-Window Protection
Sometimes you might want to protect more than just DXGI-presented content. In that case, you can actually make use of a similar mechanism to simply protect an entire window from being captured by various screen-capture techniques. It's an API called SetWindowDisplayAffinity
. It's equivalent in strength and function to passing the DXGI_SWAP_CHAIN_FLAG_DISPLAY_ONLY
flag for a swap-chain, but it also protects content presented with GDI, older DirectX technologies, etc. Pass the WDA_MONITOR
flag to enable protection for a given HWND, or pass WDA_NONE
to disable protection.
For full disclosure, I work at Microsoft on the team that handles display logic, including some forms of screen capture and content protection.