Extract (Raw or Encoded) Video Frames from any GPU Memory directly for Current Program?

Question

I have been working on Camera Video Frame Capturing for the past year. I am trying to make a video streaming plugin using webrtc and its associated technologies. I know that there are some certain plugins already in the market to stream video i.e. PixelStreaming.

Recently, I came up with new research which is required in order to perform the following;

As the latency is the key factor in order to develop more reliable streaming plugins. So, I was trying my best to optimize every process running during Camera Video Frame Capturing. I am also trying to reduce the number of extra hops during this procedure.

---

This current system which I have developed is working 100 percent fine without any crash as you can see here;

My concerning point here is that Is there any possibility that I can extract camera video frames either RAW or Encoded from GPU directly?

As I want to run Camera using GPU i.e. Nvidia GPU (GTX GEFORCE) and I can get some details using a Python module for getting the GPU status from NVIDIA GPUs using Nvidia-SMI programmatically in Python. But It does not give me any access to get, fetch or extract video frames of that Camera running on GPU.

---

Simple Example:

As an example, let's take an example of a simple camera and its frames being processed by GPU instead of CPU such as;

I have visited this pretty cool answer:

• Lowest overhead camera to CPU to GPU approach on android

As far as the Nvidia Encoder is concerned, I have also worked with this pretty cool encoder and successfully managed to encode raw video frames using NvEnc. I am also aware of its C/C++ implementation directly.

Let's take another very simple example. I have tried to allocate memory in GPU using cudaMalloc CUDA C++ Programming. I am also able to copy allocated GPU memory into Host Program using cudaMemcpy such as;

int main()
{
    const unsigned int N = 1048576;
    const unsigned int bytes = N * sizeof(int);
    int *h_a = (int*)malloc(bytes);
    int *d_a;
    cudaMalloc((int**)&d_a, bytes);

    memset(h_a, 0, bytes);
    cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);
    cudaMemcpy(h_a, d_a, bytes, cudaMemcpyDeviceToHost);

    return 0;
}

Is there any plugin, module, library, or some piece of advice someone can give me about extracting frames directly from GPU without taking frames from Camera every time for the current running process/program.?

Stay