How to use hardware video scalers?

Question

Modern graphics cards have hardware video scalers, for example as part of AMD Avivo, NVIDIA PureVideo or Intel ClearVideo. For example, AMD's Avivo whitepaper says:

"The image output scalers support up to 6 vertical filter taps and up to 10 horizontal filter taps. These scalers are high-precision polyphase scalers that are highly programmable; they are suitable for upscaling by practically any ratio, or for downscaling by up to 4:1."

The question: How can the video scaler hardware be used from a Windows program?

Assume there already exists a decoded video frame, for example in a IDirect3DSurface9, and the goal is to display that video frame on screen using the hardware scaler. I would like to use a Windows API like Media Foundation or DirectShow, rather than vendor-specific APIs if possible. I am mainly interested in upscaling by a fairly large factor around 1.5-3x.

A secondary question is, how can the video scaler hardware parameters be controlled? (For example, the filter coefficients in the polyphase filters mentioned above)

EDIT: Bounty started. Please provide an example of any way to use video scaler hardware in video card (this may be vendor specific, or use any version of DirectX/DirectShow/Media Foundation API).

EDIT: Update: Some examples of programs that do use the video scaler hardware: WinDVD, PowerDVD, madVR. I want to know how to accomplish what they do, which is to use the GPU's builtin video hardware scaler, not a scaler implemented using D3D shaders and texture samplers.

Waylin

Answer 1

Some of the possible approaches are:

1. Use MFCreateVideoRenderer to create an EVR media sink, and call IMFVideoDisplayControl::SetRenderingPrefs with MFVideoRenderPrefs_AllowScaling flag set (or use IMFAttributes and set the EVRConfig_AllowScaling attribute) and then call IMFVideoDisplayControl::SetVideoPosition to define how the result is scaled. This is part of the Enhanced Video Renderer (EVR).

2. Use IDirectXVideoProcessor::VideoProcessBlt and set DXVA2_VideoProcessBltParams::ConstrictionSize to define how the result is scaled. This is also based on EVR/DXVA.

3. (suggested by ananthonline) Use Video Resizer DSP and use IWMResizerProps::SetFullCropRegion (or MFPKEY_RESIZE_DST_WIDTH and MFPKEY_RESIZE_DST_HEIGHT) to scale the result. This is both a DirectX Media Object (DMO) and Media Foundation Transform (MFT). Note: A video MFT has the attribute MF_SA_D3D_AWARE which can be used to query whether it supports DirectX 3D hardware acceleration, and this can be enabled by sending it the MFT_MESSAGE_SET_D3D_MANAGER message.

4. Use Video Processor MFT and set IMFVideoProcessorControl::SetConstrictionSize to scale the result. This is a MFT.

5. Use a DirectX 3D device and call StretchRect to scale a surface. Note: this pretty obviously does not use the video scaler hardware, it uses texture sampler hardware. A texture can be rendered on a quad with similar effect.

I am still not sure which, if any, of these approaches uses the video scaler hardware. It is likely that at least approaches 1 and 2 would, because they are tied directly to EVR/DXVA; approaches 3 and 4 also might if they are accelerated by DXVA. A definitive answer is still needed, ideally with a reference to documentation and/or a code sample.

Waylin

Thanks! Direct3D with method 2 (DXVA2_VideoProcessBltParams::ConstrictionSize) works for me! – Lithography

Answer 2

To just access hardware - Direct2D (Direct3D is probably a better choice) is fine. But to access the video scaler you'll need to use DirectShow or Windows Media Foundation (I haven't really used this a lot).

AFAIK - the "Resizer DMO" filter will access the correct hardware and perform the video resize for you. Here is a code sample that uses this filter.

EDIT: If you are using MFTs, the Video Processor Filter also ought to be as efficient. As indicated in the documentation, "The video processor supports GPU-accelerated video processing, using Microsoft Direct3D 11. For more information, see MF_SA_D3D11_AWARE."

Also - see this section on hardware MFT attributes, especially MFT_ENUM_HARDWARE_URL_Attribute that lets you check if that MFT is implemented in hardware or not. Note that DXVA integrates with Media Foundation and exposes its functionality as MFTs. So an MFT that is implemented in hardware is most likely using the underlying hardware and is the method used by Windows itself.

Hope this helps!

Elevated

This is promising. The "Video Resizer DSP" docs are here: msdn.microsoft.com/en-us/library/windows/desktop/… This is available as either a DMO or MF Transform. The only option (other than input and output width, crop, etc) is MFPKEY_RESIZE_QUALITY which can be "high" or "low". How do we know this uses the hardware? How does this differ from VideoProcessBlt in terms of underlying implementation? – Waylin

Look at the section above that points to MFT_HARDWARE_URL_Attribute. That will indicate if the particular MFT is implemented in hardware. As to specifically accessing the particular video scaler - you're probably going to have to write code that is device specific and perhaps need proprietary driver info. This is probably the best you can do from a hardware-agnostic high-level API. – Elevated

awarded bounty because you suggested some new approaches. – Waylin

i had tried video processor filter that using direct3d11, my test graphic card is cheap intel-based which only support direct3d 10.1. and must support windows 7 platform, so only solution seems to be directshow. – Lithography

Answer 3

Direct2D is Microsoft's way of gaining access to the graphics hardware, albeit indirectly. The Direct2D Transform does scaling. You won't have any direct control over the video scaler but must trust the driver to pick the correct settings for you.

Hexone

@AlexI, the links I provided speak specifically of using the graphics hardware but unfortunately don't go into details. In fact it probably uses the 3D capabilities rather than the video specifically since it's layered on Direct3D. I'm not sure of the graphics hardware though, could it be that the same pathways are used for both purposes? – Hexone

I believe they are not the same. The video scalers (according to various spec sheets) use a multitap filter with far more taps than anything available in the 3D texture sampler, so they are probably a distinct hardware unit. – Waylin

@AlexI - I checked and you're right. Direct2D would be like using a texture sampler to do the resize (not using the built-in hardware components for scaling, decoding, etc). – Elevated