How to draw text into CIImage?

Question

How to draw into CIImage (or maybe into CVPixelBuffer, but I guess it easier to add text to CIImage)? not to UIImage

I record video (.mp4 file) using AVAssetWriter and CMSampleBuffer (from video, audio inputs). While recording I want to add text on the video frames, I'm already converting CMSampleBuffer to CIImage, now I need somehow add text to CIImage and convert back to CVPixelBuffer. I didn't really find any simple examples in Swift how to add (draw) text to CIImage or add anything else (like lines, rects, not just text).

/// encode video buffer
func appendVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {

    // before encoding video frames into a file I would like process it
    // 1. convert CMSampleBuffer to ciImage

    let cvPixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
    let ciimage : CIImage = CIImage(cvPixelBuffer: cvPixelBuffer)

    // 2. do something with CIImage


    // 3. convert CIImage back to CMSampleBuffer

    let sampleBufferWithText == ...


    appendSampleBuffer(sampleBufferWithText, ofMediaType: AVMediaType.video.rawValue)
}

/// encode audio buffer
func appendAudioSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
    appendSampleBuffer(sampleBuffer, ofMediaType: AVMediaType.audio.rawValue)
}

Bronk

@dfd then maybe somehow add text directly to CVPixelBuffer/CMSampleBuffer? – Bronk

Good, then the confusion was on my part. :-) I haven't worked much with buffers, but for a CIImage the concept is to create a UILabel, then using a UI graphics context turn it into an image, getting a CIImage from it, and then a CIFilter to merge with it. In Core Image for Swift chapter 2 Simon Gladman describes using CIHieghtFieldFromMask and CIShadedMaterial to do something close to what I'm talking about.... – Nutmeg

Here's the specific playground for it: github.com/FlexMonkey/CoreImageForSwiftPlaygrounds/tree/master/… Also, you might want to check out other projects he has, including one for live camera filtering. Most are written in Swift 2 though, so you'll either need Xcode 8 or know how to manipulate Xcode 9 project settings to upgrade things to run them. His techniques are the important part though, and they still work. – Nutmeg

@dfd but I think performance won't be that good (with UILabel), I also experimented with UIImage (which I don't use) #28907414 and adding text to it takes a lot of cpu time – Bronk

@dfd if creating CIImage from a text and blending it with frame CIImage using CIBlendWithMask will be fast then would be great, but problem how to convert CIImage back to CMSampleBuffer? – Bronk

@dfd yes combining two CIImage works pretty fast, but have to find out now how to send buffer from new combined CIImage to CMSampleBuffer – Bronk

Answer 1

You can create NSAttributedString for text -> draw this string in to cgContext -> convert UIImage/CGImage -> CIImage

Atilt