Programmatically grab screenshots in OSX

Question

I am going to port some screenshot grabbing code (C++) for linux to osx. The current solution run graphical applications in xvfb and then use xlib to grab screenshots from the display. (That will also support if we are running without xvfb).

So as I understood osx is moving away from X11 so my question is what to use besides xlib to implement it now ? I have found Quartz Display Services. Is that what makes sense to use now ? Will that work with xvfb ?

Totality

Answer 1

Yes, you will be able to call functions like CGDisplayCreateImage (documentation linked for you) by linking the Application Services framework to your C++ tool.

Marionmarionette

Can it screenshot xvfb and will xvfb even exist in future versions ? – Totality

I do not know for 100% certain, but that particular function's description says "Returns an image containing the contents of the specified display." So my read on it is that whatever is on the screen, whether xvfb or Safari or Photoshop or whatever, it'll end up in the captured image buffer. – Marionmarionette

I can add that screenshotting Xvfb did not work. So maybe there is another framebuffer that can be used on mac. But for the time I can manage without one. – Totality

I am using CGDisplayCreateImage(CGMainDisplayID()) to capture screen. When we do a fast user switch to another user, it still captures screen from first user. CGMainDisplayID() on second user returns 1104977152. Any idea why this is happening? – Covenanter

that sounds like a separate, and new question @SeemaKadavan – Marionmarionette

posted new query - #31476156 – Covenanter

Answer 2

I have written an example for capturing the pc display screen and convert to opencv Mat.

#include <iostream>
#include <opencv2/opencv.hpp>
#include <unistd.h>
#include <stdio.h>
#include <ApplicationServices/ApplicationServices.h>

using namespace std;
using namespace cv;

int main (int argc, char * const argv[])
{
    size_t width = CGDisplayPixelsWide(CGMainDisplayID());
    size_t height = CGDisplayPixelsHigh(CGMainDisplayID());

    Mat im(cv::Size(width,height), CV_8UC4);
    Mat bgrim(cv::Size(width,height), CV_8UC3);
    Mat resizedim(cv::Size(width,height), CV_8UC3);

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef contextRef = CGBitmapContextCreate(
                                                    im.data, im.cols, im.rows,
                                                    8, im.step[0],
                                                    colorSpace,    kCGImageAlphaPremultipliedLast|kCGBitmapByteOrderDefault);

    while (true)
    {
        CGImageRef imageRef = CGDisplayCreateImage(CGMainDisplayID());
        CGContextDrawImage(contextRef,
                           CGRectMake(0, 0, width, height),
                           imageRef);
        cvtColor(im, bgrim, CV_RGBA2BGR);
        resize(bgrim, resizedim,cv::Size(),0.5,0.5);
        imshow("test", resizedim);
        cvWaitKey(10);
        CGImageRelease(imageRef);
    }

//    CGContextRelease(contextRef);
//    CGColorSpaceRelease(colorSpace);

    return 0;
}

and then, the result is here.

I had expected my current display would be captured, but only the back wallpaper was captured actually. What the CGMainDisplayID() refers would be a hint to this problem.

Anyway, I hope this may approach your goal a bit.

Jennifer

Answer 3

void captureScreen(){
    CGImageRef image_ref = CGDisplayCreateImage(CGMainDisplayID()); 
    CGDataProviderRef provider = CGImageGetDataProvider(image_ref);
    CFDataRef dataref = CGDataProviderCopyData(provider);
    size_t width, height;    width = CGImageGetWidth(image_ref);
    height = CGImageGetHeight(image_ref); 
    size_t bpp = CGImageGetBitsPerPixel(image_ref) / 8;
    uint8 *pixels = malloc(width * height * bpp);
    memcpy(pixels, CFDataGetBytePtr(dataref), width * height * bpp);
    CFRelease(dataref); 
   CGImageRelease(image_ref); 
   FILE *stream = fopen("/Users/username/Desktop/screencap.raw", "w+");
   fwrite(pixels, bpp, width * height, stream);
   fclose(stream); 
   free(pixels);
}

or in C#:

// https://mcmap.net/q/1230029/-capture-screen-image-in-c-on-osx
// https://github.com/Acollie/C-Screenshot-OSX/blob/master/C%2B%2B-screenshot/C%2B%2B-screenshot/main.cpp
// https://github.com/ScreenshotMonitor/ScreenshotCapture/blob/master/src/Pranas.ScreenshotCapture/ScreenshotCapture.cs
// https://screenshotmonitor.com/blog/capturing-screenshots-in-net-and-mono/
namespace rtaStreamingServer
{

    // https://github.com/xamarin/xamarin-macios


    // https://qiita.com/shimshimkaz/items/18bcf4767143ea5897c7
    public static class OSxScreenshot
    {

        private const string LIBCOREGRAPHICS = "/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics";

        [System.Runtime.InteropServices.DllImport(LIBCOREGRAPHICS)]
        private static extern System.IntPtr CGDisplayCreateImage(System.UInt32 displayId);

        [System.Runtime.InteropServices.DllImport(LIBCOREGRAPHICS)]
        private static extern void CFRelease(System.IntPtr handle);


        public static void TestCapture()
        {
            Foundation.NSNumber mainScreen = (Foundation.NSNumber)AppKit.NSScreen.MainScreen.DeviceDescription["NSScreenNumber"];

            using (CoreGraphics.CGImage cgImage = CreateImage(mainScreen.UInt32Value))
            {
                // https://mcmap.net/q/1780174/-get-pixel-from-the-screen-screenshot-in-max-osx-duplicate/17343305#17343305

                // Get byte-array from CGImage
                // https://gist.github.com/zhangao0086/5fafb1e1c0b5d629eb76

                AppKit.NSBitmapImageRep bitmapRep = new AppKit.NSBitmapImageRep(cgImage);

                // var imageData = bitmapRep.representationUsingType(NSBitmapImageFileType.NSPNGFileType, properties: [:])
                Foundation.NSData imageData = bitmapRep.RepresentationUsingTypeProperties(AppKit.NSBitmapImageFileType.Png);

                long len = imageData.Length;
                byte[] bytes = new byte[len];
                System.Runtime.InteropServices.GCHandle pinnedArray = System.Runtime.InteropServices.GCHandle.Alloc(bytes, System.Runtime.InteropServices.GCHandleType.Pinned);
                System.IntPtr pointer = pinnedArray.AddrOfPinnedObject();
                // Do your stuff...
                imageData.GetBytes(pointer, new System.IntPtr(len));
                pinnedArray.Free();

                using (AppKit.NSImage nsImage = new AppKit.NSImage(cgImage, new System.Drawing.SizeF(cgImage.Width, cgImage.Height)))
                {
                    // ImageView.Image = nsImage;
                    // And now ? How to get the image bytes ? 

                    // https://theconfuzedsourcecode.wordpress.com/2016/02/24/convert-android-bitmap-image-and-ios-uiimage-to-byte-array-in-xamarin/
                    // https://mcmap.net/q/1780175/-nsimage-from-byte-array
                    // https://mcmap.net/q/1780176/-nsimage-source-from-byte-array-cocoa-app-xamarin-c
                    // https://gist.github.com/zhangao0086/5fafb1e1c0b5d629eb76
                    // https://www.quora.com/What-is-a-way-to-convert-UIImage-to-a-byte-array-in-Swift?share=1
                    // https://mcmap.net/q/648941/-converting-uiimage-to-byte-array

                } // End Using nsImage 

            } // End Using cgImage 

        } // End Sub TestCapture 


        public static CoreGraphics.CGImage CreateImage(System.UInt32 displayId)
        {
            System.IntPtr handle = System.IntPtr.Zero;

            try
            {
                handle = CGDisplayCreateImage(displayId);
                return new CoreGraphics.CGImage(handle);
            }
            finally
            {
                if (handle != System.IntPtr.Zero)
                {
                    CFRelease(handle);
                }
            }
        } // End Sub CreateImage 


    } // End Class OSxScreenshot 


} // End Namespace rtaStreamingServer 

Jeminah