Setting larger GOP size in MediaFoundation hardware MFT

Question

I'm trying to live stream the desktop that's captured through Desktop duplication API. H264 encoding works fine, except the fact that the Desktop duplication API delivers frames only when there is a screen change, but video encoders expect the frames to be delivered at a constant frame rate. So, I'm forced to save the previous sample to feed the encoder at a constant rate when there is no screen change triggered. This works, I could see live output at the other end.

One problem though, the encoder produces a large sample equal to the size of a fresh full-screen sample (which is probably a key-frame) at a constant rate. I have also noticed that an I frame (That large sample) is produced exactly once every 1 second (I guess, it could possibly the default GOP size) even when there is no screen change and I'm providing only the sample that I previously created and literally no diff between them except the sample time I'm setting. This is costly for a live stream, I'm not expecting the decoder to be able to seek or join the stream at the midst of the stream (At least, I have control over it), is there a way to get around this by setting a larger GOP?

I tried all the below settings, but nothing seems to be changing.

FPS: 30

CHECK_HR(pMFTOutputMediaType->SetUINT32(CODECAPI_AVEncMPVGOPSize, 1024), "Failed to set GOP size");
CHECK_HR(pMFTOutputMediaType->SetUINT32(CODECAPI_AVEncMPVGOPSInSeq, 1024), "Failed to set GOPInSeq");
CHECK_HR(pMFTOutputMediaType->SetUINT32(MF_MT_MAX_KEYFRAME_SPACING, 1024), "Failed to set keyframe spacing");

I have tried setting CODECAPI_AVEncCommonRealTime property too, are these settings incompatible with each other?

I have also tried the below code (copied from chromium https://github.com/chromium/chromium/blob/master/media/gpu/windows/media_foundation_video_encode_accelerator_win.cc). There is still no change in the keyframes count, it still generates an I frame every one second once. I think I'm missing something.

Here is the code review link for the commit in chromium that contains their discussion regarding this particular configuration. Reading this discussion gave me some hope, but no luck yet.

void SetEncoderModes() {

    VARIANT var = { 0 };

    if (!mpCodecAPI) {
        CHECK_HR(_pTransform->QueryInterface(IID_PPV_ARGS(&mpCodecAPI)), "Failed to get codec api");
    }

    var.vt = VT_UI4;
    var.lVal = 1024;

    CHECK_HR(mpCodecAPI->SetValue(&CODECAPI_AVEncMPVGOPSize, &var), "Failed to set GOP size");
}

Any help would be appreciated.

Bowshot

I think you can try different values for the encoder bitrate instead. – Hampstead

Changing the bitrate might not work. If bitrate is reduced, it would still produce the same amount of I frames at a constant rate, producing a lower quality video output which is undesirable. – Bowshot

I have also tried this code from chromium (chromium-review.googlesource.com/c/chromium/src/+/738916/3/…), but nothing seems to be changing. – Bowshot

Answer 1

The code snippet extracted from Chromium is the way to do it: you have to use ICodecAPI interface.

As documented on MSDN:

Certified Hardware Encoder

[...]

The following is the set of required and optional ICodecAPI properties for encoders to pass the HCK encoder certification.

The following Windows 8 and Windows 8.1 ICodecAPI properties are required:

[...]

CODECAPI_AVEncMPVGOPSize

So you will have the property present in most cases.

Note that you might need to set the property before you start actual streaming.

Clean

That worked. It seems it's important to set this property before SetOutputType. I was doing it the other way, assuming it would work as I was starting the encoder only after that call. – Bowshot

Looks like this setting is not honored by intel encoder. Perfectly working fine on Nvidia though. #59051943 – Bowshot

I failed to test it on Intel machines. It works perfectly on Nvidia devices. It was my mistake to assume it would behave the same on all hardware. I have experimented a lot, but unable to figure out the reason. – Bowshot

I would assume it generally works, but the value 16384 might be out of bounds. – Clean

I tried setting 256 too, but nothing seems to be changing on Intel hardware. In chromium project they have actually configured it with INT32_MAX. Even setting INT32_MAX works fine on Nvidia machines though. – Bowshot

As per my knowledge reducing GOP even further is not a good choice, I guess the default GOP itself is 128. Is there a way to check whether the encoder supports a particular range of values for GOP? – Bowshot

"but nothing seems to be changing on Intel hardware" might also mean that you do something wrong. I see no confirmation that the property has no effect with Intel's MFT. It has to be supported because of certification requirements. – Clean

I might be I'm doing something wrong, I too think the same. I have only followed the documented settings in Microsoft site and your blog(alax.info/blog/1586) though. I have also posted my exact encoder configurations in the following thread #59051943. I don't think any other configuration is incompatible with GOP setting. – Bowshot

Looks like there is some problem with other encoder configurations. I'm just playing around with the parameters on trial and error basis. Setting CODECAPI_AVEncCommonRateControlMode with eAVEncCommonRateControlMode_UnconstrainedVBR actually reduces the idle time bitrate down to less than 6KBps, but just for few seconds (3 to 8 seconds) after stream start, and again it goes back to the same story. I have updated the same in my thread also. – Bowshot

Some miracle has happened. I accidentally changed my primary monitor to a different one on my machine, now the problem is gone. Switching back to the previously selected primary monitor leads to the same problem. I suspect the d3ddevice to be the trouble maker. Not sure yet, have to experiment a bit. – Bowshot