iOS: is Core Graphics implemented on top of OpenGL?

Question

I have found one diagram that shows Core Graphics implemented above OpenGL, and another that puts it alongside OpenGL. I would think that Apple would be smart to give each equal access to the graphics hardware but then again, I don't know much about the graphics chip they are using... maybe it is 3D all the way?

Does anybody here know the specifics?

Paroicous

Answer 1

Yes, on iOS Core Graphics (Quartz) appears to be layered on top of OpenGL ES for drawing that targets the screen, although not in an explicit way that we have access to.

Core Graphics takes vector elements (lines, arcs, etc.) and some raster ones (images) and processes them for display to the screen or for other forms of output (PDF files, printing, etc.). If the target is the screen on iOS, those vector elements will be hosted in a CALayer, either directly or through the backing layer of a UIView.

These Core Animation layers are effectively wrappers around rectangular textures on the GPU, which is how Core Animation can provide the smooth translation, scaling, and rotation of layers on even the original iPhone hardware. I can't find a reference for it right now, but at least one WWDC presentation states that OpenGL ES is used by Core Animation to communicate with the GPU to perform this hardware acceleration. Something similar can be observed on the new dual-GPU MacBook Pros, where the more powerful GPU kicks in when interacting with an application using Core Animation.

Because Core Graphics rasterizes the vector and raster elements into a CALayer when drawing to the screen, and a CALayer effectively wraps around an OpenGL ES texture, I would place OpenGL ES below Core Graphics on iOS, but only for the case where Core Graphics is rendering to the screen. The reason for the side-by-side placement in the hierarchy you saw may be due to three factors: on the Mac, not all views are layer-backed, so they may not be hardware accelerated in the same way; we can't really interact with the OpenGL ES backing of standard UI elements, so from a developer's point of view they are distinct concepts; and Core Graphics can be used to render to offscreen contexts, like PDF files or images.

Resplendent

That's really cool. In the past there was an effort to implement an X server on top of OpenGL... not sure how well that fared. There is also a widget set that works on top of OpenGL, called GLUI, that I have used too good effect. Apple as always has taken the extra effort to make the best product. – Paroicous

I'm having trouble believing this. With CG-drawing test app (on iOS 6), I'm seeing performance well below what a GL-backed system should provide (filling a stroking a few large paths gives me 12 FPS on an iPhone 4S), the OpenGL ES Analyzer instrument reports “No GL data”, and 63% of the CPU time (in this CG demo app) is being spent in CG's argb32_mark and blitting subfuctions and another 23% in aa_render. If there is GL in here, it's not being used for really anything related to drawing. – Vig

For the argument that CG and CA use GL for composition— the affine transforms I could see, but that's it. I wouldn't call it layering CG on GL; for CG to be “layered” on GL its functionality would have to be an abstraction of what GL does, but still using only GL. It seems for a large portion of CG, this is not true. For that matter, all drawing in iOS ends up on a GL poly before hits the screen— this is how Apple's been able to add the multitasking bar and notification center handle to ever app without issues. But that doesn't mean the drawing routines in those apps are using GL underneath. – Vig

@SlippD.Thompson - Yes, Core Graphics does all its rasterization CPU-side, and only uploads the end bitmap to a texture. I tried to express that, but perhaps my wording wasn't the clearest. On the Mac, they're still experimenting with Quartz Extreme and its use of OpenGL to do the actual drawing, but I don't know if we'll see that appear on iOS. – Resplendent

Answer 2

Core Graphics and OpenGL are two completely separate systems. Look at the image below (source), which shows both listed at the same level. The description of Core Graphics lower on the same page also indicates that it is the lowest-level native drawing system.

Also see the About OpenGL ES page. It shows that the OpenGL code runs directly on the GPU, and if you scroll down you will see that there are some things which cannot be done with an application that uses OpenGL. Obviously, if CG was based on OpenGL, you wouldn't be able to do those things ever.

Finally, look at the Drawing Model page for iOS. At the top, it compares OpenGL to native drawing, indicating that they work separately from each other.

Benzo

The first image/diagram you point to refers to Mac OS/X, not iOS. – Paroicous

On iOS, I don't believe this is true. As I point out in my answer, every display element on iOS is layer-backed, and these layers are GPU-accelerated, so anything that draws into them (like Core Graphics) should be layered on OpenGL ES. On the Mac, this is not always true, which would explain the independence of Quartz and OpenGL on that platform. – Resplendent

Think about how ironic this is. The reason why Macs & PCs and X11 uses compositing rather than 2D on top of 3D, is that doing so uses less resources. It really makes no sense to use 3D hardware to display something like a 2D xterm window. Yet on the far more resource restricted iOS devices Apple has taken a more-resource-hungry approach. – Paroicous

@BradLarson 1) Core graphics does not have to draw into a view. You can create a context which never interacts with core animation. 2) The layer isn't responsible for displaying the content. All it does is transform/animate a core graphics context before the view displays it. If no transformations/animations are necessary, it doesn't have to do anything. – Benzo

My point is that, while the view system on iOS may use OpenGL, Core Graphics does not. – Benzo

@Benzo - You're correct about the offscreen rendering that can be performed by Core Graphics, so I've amended my answer to make this more explicit. In the case of CALayers, they wrap around rasterized bitmaps of the Core Graphics drawing (I had to write my own recursive rendering routines to make sure that a layer hierarchy is converted to vectors on output to PDF). Even if no transforms are applied, they do direct the compositing of visual content, which has to occur on the GPU. – Resplendent

@BradLarson I think your definition of “every display element” is too broad. From my inspection, it seems CocoaTouch elements (namely, UIViews and derivatives) are CALayer-backed, while CoreGraphics draw calls are not. An example stacktrace showing this: cl.ly/N7Ss – Vig

Answer 3

Core Graphics and OpenGL are separate technologies. UIKit and AppKit are built on top of both, as well as Core Animation. You can see the graphics technology stack inside Apple's documentation (Core Animation Programming Guide)

Pu

Answer 4

As of iOS 9 Core Graphics on iOS are based on Apple's Metal framework, not OpenGL.

Stirps