Efficiently read the average color of the screen content rendered by XBMC

Question

I want to get the average color of the screen content when running XBMC to change the color of a TV ambient light. XBMC is running on a small HTPC with OpenGL ES 2.0 hardware (Raspberry Pi) running a Debian-derived distribution. I guess I have to read from the screen frame buffer in which XBMC draws using OpenGL. (At least, I think and hope that XBMC renders everything using OpenGL.)

Is it possible to read the OpenGL frame buffer representing the whole screen output? What am I going to need to access it? Do I also need an own render context to access the frame buffer of the screen? (I don't render anything by myself to the screen, I only want to read).

Efficiently calculating the average color is the next task. I'm thinking about reading every 8th or 16th pixel per row/column (would be enough, we are talking about 1080p HD movies) and then calculating the average on the CPU. Any ideas for a better solution are welcome.

Twannatwattle

Frame buffer, not screen buffer, because you asked. Both are correct though. – Rapper

@Rapper Thanks. I thought about saying "frame buffer", however, the reader might think that I want to read back something I have rendered into before. Which is not the case here. I'll just emphasize that it's the whole screen. – Twannatwattle

The frame buffer contains exactly what is currently on the screen. That's why I say that. – Rapper

@Rapper Hm, please correct me again if I'm wrong, but I thought I can create a frame buffer for off-screen rendering. So in general, a frame buffer is not the screen content. Plus we can't talk about "the" frame buffer. Or am I misunderstanding something here? Or is "frame buffer" != "frame buffer object"? – Twannatwattle

We're talking about the frame buffer internally kept by the kernel and graphics card driver, which is usually under the control of X11, which manages smaller frame buffers for window managers and applications. – Rapper

have you considered using libavfilter instead? (xbmc uses ffmpeg under the hood for decoding - libva accelerated in this case) – Clifton

@Clifton Indeed, this sounds like an alternative, thanks for this hint. However, I'm prefering a solution working independently from the type of screen content (as long as it is drawn with OpenGL), so it also works if - for what reason ever - I watch a movie with something different than XBMC, or maybe even play a game (OpenGL-based). Long story short: If it is possible with "good" performance, I prefer an OpenGL solution. Otherwise your idea seems to be the best alternative. – Twannatwattle

Hi @leemes, have you by any chance come to a solution for this? I stumbled upon the same challenge myself and currently researching ways of capturing the screen contents from OpenGl. There are tools out there like glc and SimpleScreenRecorder that implement the functionality we're after, but I havent tried any of them yet. – Martel

@Martel Sadly I hadn't have time to work on this. But I'm still very interested in a solution, since my TV is still missing a fancy dynamic back light which I wanted to control using this methd... :( So if you have any news, it would be cool if you could let me know! But hey, this project "glc" seems pretty promising. Do you also want to read the average color only? Or full frames? – Twannatwattle

Answer 1

You should take a look at the source code of Boblight.

Extending Boblight seems like a viable alternative (if it does not support what you need already!).

If not, look at src/clients/ folder. boblight-getpixel.c (for MS Windows) and boblight-X11.c are 'pixel grabbers'; standalone programs that do exactly what you need, and then communicate the grabbed color to the boblight server.

In boblight-X11.c you have examples of using XShmGetImage or the slower XGetImage to read portions of the screen using X11/extensions/XShm.h, a portion of that code does:

[...]

if(!XShmGetImage(dpy, root_win, xim, 0, 0, AllPlanes))
{
   sleep(1);
   pthread_mutex_unlock(&grabmutex);
   return;
}

XSync(dpy, True);

for (x = 0; x < width; x += xadd)
{
    for (y = 0; y < height; y += yadd)
    {
       pixel = XGetPixel(xim, x, y);
       RGB[0] = (pixel >> 16) & 0xff;
       RGB[1] = (pixel >>  8) & 0xff;
       RGB[2] = (pixel >>  0) & 0xff;

       /*add it to the RGB array*/
       boblight_add_pixel (&config, RGB, x, y);
   }
}

[...]

Britt

Thank you! Very interesting. Indeed, maybe I can just use this program mainly unchanged to color my lights! I'm wondering if this approach is efficient, since I use a low-end hardware (Raspberry Pi). I think I just have to try this. Thank you again for this pointer, which I couldn't find on google (maybe I was too focused on OpenGL) – Twannatwattle

Although XMBC uses OpenGL for it's UI, I think it has many different options regarding video output. Don't forget to accept the answer when you see fit ;) – Britt

The links did not age well. – Peltast

@Peltast Thanks, fixed that. But wait until Wayland becomes ubiquitous... none of this will make any sense ;-) – Britt