How does GUI output work from application to hardware level?

Question

I am getting into GUI programming and did some research. Now not everything is clear to me. If I would use GTK+ as toolkit, how does it communicate with the graphics card?

On a Linux system I suppose it would be GTK --> X Server --(OpenGL)--> graphics card. Is this right?

I read that some GUIs directly draw OpenGL (e.g. Blender3D), so how do other apps draw their GUIs?

If the only APIs (that i know of) for graphics cards is Direct3D and OpenGL, what is the distinction between software rendering and hardware acceleration?

Can software that does "software rendering" directly write to the framebuffer of the graphics card, so that OpenGL is untouched?

PS: sorry for the many questions, but i don't really get it how that all works, thanks for every answer :)

Ozonolysis

Answer 1

On a Linux system I suppose it would be GTK --> X Server --(OpenGL)--> graphics card. Is this right?

No. GTK+ on Linux goes

                                              /-[ if direct context ]---\
                     /--> OpenGL >-+---------/                           \
                    /              \-> GLX -+--------------\              \
                   /                         \              \              \
GTK+ -+-> cairo >-+---> XRender >--------+----+-> Xlib/xcb >-+-> X server >-+-> Kernel Module >-> GPU
       \           \–-> pixmap buffer >–/
        \                              /
         \―---------------------------/

I read that some GUIs directly draw OpenGL (e.g. Blender3D), so how do other apps draw their GUIs?

It's just "Blender" (no trailing 3D). Blender's GUI toolkit uses OpenGL as its only backend, yes. But the GUI is not directly drawn using OpenGL, that would be just to cumbersome to work with (draw each and every button using OpenGL calls. Blender has its very own toolkit. GTK+ is another toolkit but not tied to Blender (in fact, one of my pet projects is extracting Blender's GUI toolkit, so that it can be used in independent projects).

Toolkits like GTK+ and Qt are designed for maximum portability. Blender has the luxury of knowing, that there will be OpenGL available. Apps developed for GTK+ or Qt may be able to run on non 3D capable systems, so the design of GTK+ and Qt allow to run on a number of backends. GTK+ now in version 3 uses the Cairo graphics engine as graphics backend. Cairo again has its own backends, namely a software rasterizer drawing into pixmaps (pixel images), or proxying drawing commands to an underlying graphics architecture. In the case of Cairo on Linux this may be either OpenGL or X11 (core and XRender protocol extensions).

If the only APIs (that i know of) for graphics cards is Direct3D and OpenGL, what is the distinction between software rendering and hardware acceleration?

Neither OpenGL nor Direct3D talk to the graphics card. They talk to the graphics card's drivers. So the option you'd have would be, talking to the drivers youself, bypassing OpenGL and Direct3D. But why would you do this? It's tedious.

On Windows in addition you have the GDI and/or the WPF (Windows Presentation Foundation) for drawing stuff, and Direct2D.

On Linux you get the X11 core and XRender extension protocol for drawing nice pictures.

Another API in the rise is OpenVG, which aims to standardize all those 2D drawing APIs. And at least in Linux OpenGL and OpenVG have been selected to become the only available abstract drawing APIs in the long term, with some windowing system for managing the framebuffer and user input. There's Wayland in development (which design I completely dislike) and X11, which I think has the better design (it's a network oriented system, that allows for distributed execution, something I consider very important in the future), but is in need of a complete overhaul into some "X12" – cleaning out the legacy cruft, make it operate in a contact color space, make connections transitionable (so that you can migrate clients between X server, which would allow for a much more elegant way of locking X sessions, by moving all the connections into some shadow X server, instead of trying to block access using a locking screen saver).

Can software that does "software rendering" directly write to the framebuffer of the graphics card, so that OpenGL is untouched?

Not on a modern operating system. However the OS may give you an abstracted access to the graphics card through some framebuffer API (/dev/fb0 on Linux). However the framebuffer is unmanaged, so if there's a X server or Wayland running, either of those are in task of managing the FB, and it's none of your business then.

Hoodoo

What is the difference between software acceleration and hardware acceleration ? – Ked

I am a beginner in GUI Development, could you refer me a book that explains everything from above to hardware level ? – Ked

Lastly , as far as i know , if i use a library in C . Then the functions in library , must be able to ultimately , break down to lowest level in C. What i mean is if i use scanf in C . It will ultimately , use any other function present in C, and step by step , everything ultimately boils down lowest level of abstraction layer we can get in C. What i want to as k is if i us cairo Library and use it functions. Can i draw gui without using any extra library. Does C natively can do gui programming? – Ked

@SurajJain: At the uttermost lowest level you're going to end up as some hand written assembly for the architecture the program is running on. But all operating systems (because that's the very task of an OS) build higher level abstraction layers around this. So there's a handful of functions, written in assembly and wrapped in a interface compatible to the systems programming language used (that can also be Rust, Go, Swift, or any other language that compiles to code that can run close to the metal), and then those functions are used for the rest. – Hoodoo

@SurajJain: I can not really recommend a single book on how the whole thing works. That being said, modern PCs are far too complex to offer a nice learning platform. If you really, really want to get into the gory details, I suggest you get yourself a nice microcontroller and a display that you can connect to it and then, from scratch write a few programs for that push pixels to the display. Then write graphics library on top of that. In the end, this is what's going on under the hood of PCs, too, just far more sophisticated. – Hoodoo

So you mean if i use any graphic library , they are function written in assembly not in the language code. What i mean is if i use a function in c it should use Language native function. So do the graphic library do not use any C Characteristics..? If i do not use any library can i still make GUI in C? – Ked

@SurajJain: No, what I mean is, that it is several layers written in high level language and only at the bottom most layer (the one that is pushing and pulling data to and from the hardware) is written in assembly. In fact most drivers are written in pretty high level languages (I've even seen a driver written in Java; not a bright idea, but definitely possible). It's only the OS that really has to get into the muddy waters of "pedal-to-the-metal" low level stuff; the whole purpose of an OS is to provide the scaffolding that more complex things can be done at a higher level. – Hoodoo

@SurajJain: If you want to know how a typical OpenGL implementation is done, look at the source code of Mesa. It's written mostly in C and C++. It talks to the kernel side using an API called DRM (Direct Rendering Manager). On the kernel side there's another (largeish) bunch of code written in C that does the kernel side DRM stuff. The C code in turn calls C code that on a high level knows how to talk to the hardware. This code in turn talks to lower level C functions. And eventually this reaches a layer where it has to actually push and pull data and the OS kernel offers functions for that. – Hoodoo

@SurajJain: These functions down at the bottom of the OS kernel are the functions that are outside the scope of C. And it's only that handfull of these functions that must be written in assembly. If you look at the Linux kernel just search for all code files ending in .S. These are the metal-level assembly code files. And they make only a very very tiny fraction of the whole OS. – Hoodoo

So you mean i can code a gui program in C , with basic C functions , till i reach the layer when i cannot use C and had to resort to functions provided by OS? – Ked

If i could write most of the layers in C . Why do i use OpenGL .? What is the point ? – Ked

So (C Functions + handful of functions, written in assembly and wrapped in a interface compatible ) = OpenGl Am I Right ? ..... what i mean is you have to use some C functions and some functions written in assembly , i can get to do the same thing as i do in OpenGl but with much workload and also not portable.... – Ked

@SurajJain: Saving yourself huge amounts of work and not having to deal with hundreds of different kinds of hardware out there. Making a Radeon draw a triangle requires different commands than making a GeForce draw the same triangle; in face different kinds of GeForce GPUs require different chains of commands. OpenGL gives you a nice, unified interface. Your programs have to know only OpenGL. And the drivers for GPUs offer OpenGL on one end and make the right calls (written in a high level language) to make the GPU do what you want on the other end. – Hoodoo

@SurajJain: OpenGL is a specifications. And lots and lots(!) of C functions specific to a single piece of hardware make an OpenGL driver for that hardware. Another piece of hardware requires a different driver, which again requires lots and lots of functions and because these are for a different GPU you can not reuse the functions of the other drivers. That's why you want a unified API. You do not want to write all the stuff yourself. It's tedious, difficult work. For example the OpenGL driver of NVidia with all its heuristics and corner cases is about 1 million LoC. – Hoodoo

Oh... now i am understanding. I was not asking all this with intention to make one , but so that i do not see opengl as some magic box , performing action, i just wanted to know , that it is not a magic box , and the workings of it. When you said "handful of functions, written in assembly and wrapped in a interface compatible to the systems programming language used" What did you meant? – Ked

@SurajJain: The respective functions act as kind of a glue between C code and the architecture specific machine instructions that are required to send commands to I/O ports of attached devices. For example every PCI card you plug into your computer has a number of I/O ports and you can control the PCI card by writing the right bit patterns to these ports. The PCI bus is connected at a very low level to the machine. For details I suggest you simply look for all files ending with .s in the Linux kernel and see what they do. – Hoodoo

ok.. i will do so. Do you recommend any text that just skim over the details of gui ? – Ked

Let us continue this discussion in chat. – Hoodoo

Answer 2

Can software that does "software rendering" directly write to the framebuffer of the graphics card, so that OpenGL is untouched?

Not quite, or more accurately: depends on the driver. Display systems like XServer (or the Windows and MacOS equivalents) don't use APIs but rather describe an API. Specifically called a driver model (or in XServers case one of multiple models like XAA, EXA, UXA and DRI) that the display driver has to satisfy.

Everything that is not defined in these models has to be done in "software" and calculated on the CPU. Additionally some of the operations defined by the models maybe calculated on the CPU by the driver as well.

OpenGL used to be completely separate standard from these models. Graphics drivers didn't have to implement OpenGL at all to be used by the OS. But with the continuing integration of 3D graphics and more complex compositing in modern OSs the driver models have started to rely on the same paradigms as OpenGL and Direct3D. For example WDDM (the Windows Display Driver Model) outright requires Direct3D 9 support.

Moe