How to keep NSBitmapImageRep from creating lots of intermediate CGImages?

I have a generative art application which starts with a small set of points, grows them outwards, and checks the growth to make sure it doesn't intersect with anything. My first naive implementation was to do it all on the main UI thread with the expected consequences. As the size grows, there are more points to check, it slows down and eventually blocks the UI.

I did the obvious thing and moved the calculations to another thread so the UI could stay responsive. This helped, but only a little. I accomplished this by having an NSBitmapImageRep that I wrap an NSGraphicsContext around so I can draw into it. But I needed to ensure that I'm not trying to draw it to the screen on the main UI thread while I'm also drawing to it on the background thread. So I introduced a lock. The drawing can take a long time as the data gets larger, too, so even this was problematic.

My latest revision has 2 NSBitmapImageReps. One holds the most recently drawn version and is drawn to the screen whenever the view needs updating. The other is drawn to on the background thread. When the drawing on the background thread is done, it's copied to the other one. I do the copy by getting the base address of each and simply calling memcpy() to actually move the pixels from one to the other. (I tried swapping them rather than copying, but even though the drawing ends with a call to [-NSGraphicsContext flushContext], I was getting partially-drawn results drawn to the window.)

The calculation thread looks like this:

    BOOL    done    = NO;
    while (!done)
    {
        self->model->lockBranches();
        self->model->iterate();
        done = (!self->model->moreToDivide()) || (!self->keepIterating);
        self->model->unlockBranches();

        [self drawIntoOffscreen];

        dispatch_async(dispatch_get_main_queue(), ^{
            self.needsDisplay = YES;
        });
    }

This works well enough for keeping the UI responsive. However, every time I copy the drawn image into the blitting image, I call [-NSBitmapImageRep baseAddress]. Looking at a memory profile in instruments, each call to that function causes a CGImage to be created. Furthermore, that CGImage isn't released until the calculations finish, which can be several minutes. This causes memory to grow pretty large. I'm seeing around 3-4 Gigs of CGImages in my process, even though I never need more than 2 of them. After the calculations finish and the cache is emptied, my app's memory goes down to only 350-500 MB. I hadn't thought to use an autorelease pool in the calculation loop for this, but will give it a try.

It appears that the OS is caching the images it creates. However, it doesn't clear out the cache until the calculations are finished, so it grows without bound until then. Is there any way to keep this from happening?

Don't use -bitmapData and memcpy() to copy the image. Draw the one image into the other.

I often recommend that developers read the section "NSBitmapImageRep: CoreGraphics impedance matching and performance notes" from the 10.6 AppKit release notes:

NSBitmapImageRep: CoreGraphics impedance matching and performance notes

Release notes above detail core changes at the NSImage level for SnowLeopard. There are also substantial changes at the NSBitmapImageRep level, also for performance and to improve impedance matching with CoreGraphics.

NSImage is a fairly abstract representation of an image. It's pretty much just a thing-that-can-draw, though it's less abstract than NSView in that it should not behave differently based aspects of the context it's drawn into except for quality decisions. That's kind of an opaque statement, but it can be illustrated with an example: If you draw a button into a 100x22 region vs a 22x22 region, you can expect the button to stretch its middle but not its end caps. An image should not behave that way (and if you try it, you'll probably break!). An image should always linearly and uniformly scale to fill the rect in which its drawn, though it may choose representations and such to optimize quality for that region. Similarly, all the image representations in an NSImage should represent the same drawing. Don't pack some totally different image in as a rep.

That digression past us, an NSBitmapImageRep is a much more concrete object. An NSImage does not have pixels, an NSBitmapImageRep does. An NSBitmapImageRep is a chunk of data together with pixel format information and colorspace information that allows us to interpret the data as a rectangular array of color values.

That's the same, pretty much, as a CGImage. In SnowLeopard an NSBitmapImageRep is natively backed by a CGImageRef, as opposed to directly a chunk of data. The CGImageRef really has the chunk of data. While in Leopard an NSBitmapImageRep instantiated from a CGImage would unpack and possibly process the data (which happens when reading from a bitmap file format), in SnowLeopard we try hard to just hang onto the original CGImage.

This has some performance consequences. Most are good! You should see less encoding and decoding of bitmap data as CGImages. If you initialize a NSImage from a JPEG file, then draw it in a PDF, you should get a PDF of the same file size as the original JPEG. In Leopard you'd see a PDF the size of the decompressed image. To take another example, CoreGraphics caches, including uploads to the graphics card, are tied to CGImage instances, so the more the same instance can be used the better.

However: To some extent, the operations that are fast with NSBitmapImageRep have changed. CGImages are not mutable, NSBitmapImageRep is. If you modify an NSBitmapImageRep, internally it will likely have to copy the data out of a CGImage, incorporate your changes, and repack it as a new CGImage. So, basically, drawing NSBitmapImageRep is fast, looking at or modifying its pixel data is not. This was true in Leopard, but it's more true now.

The above steps do happen lazily: If you do something that causes NSBitmapImageRep to copy data out of its backing CGImageRef (like call bitmapData), the bitmap will not repack the data as a CGImageRef until it is drawn or until it needs a CGImage for some other reason. So, certainly accessing the data is not the end of the world, and is the right thing to do in some circumstances, but in general you should be thinking about drawing instead. If you think you want to work with pixels, take a look at CoreImage instead - that's the API in our system that is truly intended for pixel processing.

This coincides with safety. A problem we've seen with our SnowLeopard changes is that apps are rather fond of hardcoding bitmap formats. An NSBitmapImageRep could be 8, 32, or 128 bits per pixel, it could be floating point or not, it could be premultiplied or not, it might or might not have an alpha channel, etc. These aspects are specified with bitmap properties, like -bitmapFormat. Unfortunately, if someone wants to extract the bitmapData from an NSBitmapImageRep instance, they typically just call bitmapData, treat the data as (say) premultiplied 32 bit per pixel RGBA, and if it seems to work, call it a day.

Now that NSBitmapImageRep is not processing data as much as it used to, random bitmap image reps you may get ahold of may have different formats than they used to. Some of those hardcoded formats might be wrong.

The solution is not to try to handle the complete range of formats that NSBitmapImageRep's data might be in, that's way too hard. Instead, draw the bitmap into something whose format you know, then look at that.

That looks like this:
NSBItmapImageRep *bitmapIGotFromAPIThatDidNotSpecifyFormat;
NSBitmapImageRep *bitmapWhoseFormatIKnow = [[NSBitmapImageRep alloc] initWithBitmapDataPlanes:NULL pixelsWide:width pixelsHigh:height
                                                  bitsPerSample:bps samplesPerPixel:spp hasAlpha:alpha isPlanar:isPlanar
                                                  colorSpaceName:colorSpaceName bitmapFormat:bitmapFormat bytesPerRow:rowBytes
                                                  bitsPerPixel:pixelBits];
[NSGraphicsContext saveGraphicsState];
[NSGraphicsContext setContext:[NSGraphicsContext graphicsContextWithBitmapImageRep:bitmapWhoseFormatIKnow]];
[bitmapIGotFromAPIThatDidNotSpecifyFormat draw];
[NSGraphicsContext restoreGraphicsState];
unsigned char *bitmapDataIUnderstand = [bitmapWhoseFormatIKnow bitmapData];
This produces no more copies of the data than just accessing bitmapData of bitmapIGotFromAPIThatDidNotSpecifyFormat, since that data would need to be copied out of a backing CGImage anyway. Also note that this doesn't depend on the source drawing being a bitmap. This is a way to get pixels in a known format for any drawing, or just to get a bitmap. This is a much better way to get a bitmap than calling -TIFFRepresentation, for example. It's also better than locking focus on an NSImage and using -[NSBitmapImageRep initWithFocusedViewRect:].

So, to sum up: (1) Drawing is fast. Playing with pixels is not. (2) If you think you need to play with pixels, (a) consider if there's a way to do it with drawing or (b) look into CoreImage. (3) If you still want to get at the pixels, draw into a bitmap whose format you know and look at those pixels.

In fact, it's a good idea to start at the earlier section with a similar title — "NSImage, CGImage, and CoreGraphics impedance matching" — and read through to the later section.

By the way, there's a good chance that swapping the image reps would work, but you just weren't synchronizing them properly. You would have to show the code where both reps were used for us to know for sure.

NSBitmapImageRep: CoreGraphics impedance matching and performance notes

Recommended topics

Hot tags