Capture video data from screen in Python

Question

Is there a way with Python (maybe with OpenCV or PIL) to continuously grab frames of all or a portion of the screen, at least at 15 fps or more? I've seen it done in other languages, so in theory it should be possible.

I do not need to save the image data to a file. I actually just want it to output an array containing the raw RGB data (like in a numpy array or something) since I'm going to just take it and send it to a large LED display (probably after re-sizing it).

Narcoma

https://mcmap.net/q/324138/-screen-capture-with-opencv-and-python-2-7 – Investigation

Answer 1

With all of the above solutions, I was unable to get a usable frame rate until I modified my code in the following way:

import numpy as np
import cv2
from mss import mss
from PIL import Image

bounding_box = {'top': 100, 'left': 0, 'width': 400, 'height': 300}

sct = mss()

while True:
    sct_img = sct.grab(bounding_box)
    cv2.imshow('screen', np.array(sct_img))

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        cv2.destroyAllWindows()
        break

With this solution, I easily get 20+ frames/second.

For reference, check this link: OpenCV/Numpy example with mss

Coinsurance

65fps on a Mac Mini - awesome! – Karol

Saving as a .npy file is a breeze, I am getting a frame every 10-15ms. Turns out to be 100FPS - 67FPS. I guess SSD is also playing an important role here. – Preshrunk

Awesome, this should be the top answer. – Kalli

Answer 2

There is an other solution with mss which provide much better frame rate. (Tested on a Macbook Pro with MacOS Sierra)

import numpy as np
import cv2
from mss import mss
from PIL import Image

mon = {'left': 160, 'top': 160, 'width': 200, 'height': 200}

with mss() as sct:
    while True:
        screenShot = sct.grab(mon)
        img = Image.frombytes(
            'RGB', 
            (screenShot.width, screenShot.height), 
            screenShot.rgb, 
        )
        cv2.imshow('test', np.array(img))
        if cv2.waitKey(33) & 0xFF in (
            ord('q'), 
            27, 
        ):
            break

Melentha

I get ValueError: not enough image data when using Image.frombytes – Spielman

with an i7 2600k, I only get 8fps when recording a 1440p monitor. – Rigidify

As of July 2019, this code will trigger this error: "AttributeError: 'MSS' object has no attribute 'get_pixels'". – Atencio

@Atencio To circumvent this, install an older version of mss: pip uninstall mss, pip install mss==2.0.22 – Incoordination

Another workaround is to use mss().grab(window) The code would look similar to this: screenshot = mss.mss().grab(window) img = Image.frombytes("RGB", (screenshot.width, screenshot.height), screenshot.rgb) – Kikelia

To all FUTURE readers: You have to add cv2.destroyAllWindows() at the bottom (after while / end), otherwise your window will close (when you press q (ln.7 if statement) though the window itself will not get "destroyed" / "closed". When working with cv2 and cv2.imshow, never forget cv2.destroyAllWindows() at the end of your code. If you are using jupyter notebook and your cv2 window froze because you forgot it, simply restart the kernel (Kernel -> Restart) – Sciential

the colors are different; red becomes blue and blue becomes yellow... why is that? – Savoirfaire

Answer 3

You will need to use ImageGrab from Pillow (PIL) Library and convert the capture to numpy array. When you have the array you can do what you please with it using opencv. I converted capture to gray color and used imshow() as a demonstration.

Here is a quick code to get you started:

from PIL import ImageGrab
import numpy as np
import cv2

img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height *starts top-left)
img_np = np.array(img) #this is the array obtained from conversion
frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
cv2.imshow("test", frame)
cv2.waitKey(0)
cv2.destroyAllWindows()

you can plug an array there with the frequency you please to keep capturing frames. After that you just decode the frames. don't forget to add before the loop:

fourcc = cv2.VideoWriter_fourcc(*'XVID')
vid = cv2.VideoWriter('output.avi', fourcc, 6, (640,480))

and inside the loop you can add:

vid.write(frame) #the edited frame or the original img_np as you please

UPDATE
the end result look something like this (If you want to achieve a stream of frames that is. Storing as video just a demonstration of using opencv on the screen captured):

from PIL import ImageGrab
import numpy as np
import cv2
while(True):
    img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height)
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    cv2.imshow("test", frame)
    cv2.waitKey(0)
cv2.destroyAllWindows()

Hope that helps

Westnorthwest

Not sure what you mean by: "you can plug an array there with the frequency you please to keep capturing frames"... array where? And I don't need to do video output at all... Is there a way of specifying where on the screen to grab? – Narcoma

I meant to say add a while look to have the stream of screencaptures. I will edit the code now. as for specifying specific region use bbox parameter. One min I will make the update – Westnorthwest

I made further changes. Hope that helps – Westnorthwest

This essentially uses ImageGrab.grab(), which in my experience is very very slow (about 2 fps on my Macbook Pro 2015). This solution can't capture at more than 15fps as the OP asked. I've also tried the mss library, which is much faster, but I didn't manage to guarantee a steady fps or regular intervals between images. – Messidor

ImageGrab is macOS and Windows only – Evvie

@Westnorthwest I tried above code on my mac machine. It is generating output.avi of size 0 kb. I am not sure what's wrong – Isotropic

Answer 4

based on this post and others posts, i made something like this . Its taking a screenshot and writing into a video file without saving the img.

import cv2
import numpy as np
import os
import pyautogui

output = "video.avi"
img = pyautogui.screenshot()
img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
#get info from img
height, width, channels = img.shape
# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output, fourcc, 20.0, (width, height))

while(True):
 try:
  img = pyautogui.screenshot()
  image = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
  out.write(image)
  StopIteration(0.5)
 except KeyboardInterrupt:
  break

out.release()
cv2.destroyAllWindows()

Company

Thanks man.Just two typos on line 18 you should pass img not image and on line 19 you should image not img. – Hessite

thanks,didn't noticed just wanted to put it together :D (will fix it soon) – Plano

Answer 5

You can try this code as it is working for me. I've tested it on Linux

import numpy as np
import cv2
from mss import mss
from PIL import Image

sct = mss()

while 1:
    w, h = 800, 640
    monitor = {'top': 0, 'left': 0, 'width': w, 'height': h}
    img = Image.frombytes('RGB', (w,h), sct.grab(monitor).rgb)
    cv2.imshow('test', cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

Make sure that the following packages are installed:

Pillow, opencv-python, numpy, mss

Evvie

Great approach, managed to get an average of 90 fps using this method – Vernievernier

Answer 6

You can try this=>

import mss
import numpy

with mss.mss() as sct:
    monitor = {'top': 40, 'left': 0, 'width': 800, 'height': 640}
    img = numpy.array(sct.grab(monitor))
    print(img)

Questioning

Answer 7

If anyone looking for a much easier and fastest way to grab screen as frame in python, then try ScreenGear API from my high-performance video-processing vidgear library in just few lines of python code on any machine (Tested on all platforms, including Windows 10, MacOS Serra, Linux Mint) and enjoy threaded screen-casting:

Note: It also supports multiple backends and screens out-of-the box.

# import required libraries
from vidgear.gears import ScreenGear
import cv2

# define dimensions of screen w.r.t to given monitor to be captured
options = {'top': 40, 'left': 0, 'width': 100, 'height': 100}

# open video stream with defined parameters
stream = ScreenGear(logging=True, **options).start()

# loop over
while True:

    # read frames from stream
    frame = stream.read()

    # check for frame if Nonetype
    if frame is None:
        break


    # {do something with the frame here}


    # Show output window
    cv2.imshow("Output Frame", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

# close output window
cv2.destroyAllWindows()

# safely close video stream
stream.stop()

VidGear library Docs: https://abhitronix.github.io/vidgear

ScreenGear API: https://abhitronix.github.io/vidgear/latest/gears/screengear/overview/

More examples: https://abhitronix.github.io/vidgear/latest/gears/screengear/usage/

Audre

one question to ask : which platform(s) does this library support ? thank you – Duck

@PaulLam All of them. You'll able to install it on any platform of your choice: abhitronix.github.io/vidgear/latest/installation/… – Audre

oh wow that pretty nice : ) thanks for the reply and link. Does it also support mobile OS like Android and iOS 🤔 ? – Duck

@PaulLam Someone did tried it on android: github.com/rubar-tech/vidgear-kivy-android-opencv but not sure about iOS. If you're able to install and run python on iOS then its a yes. – Audre

wow that’s smooth , I’ll try it later : ) thanks again – Duck

I tried to use and import vidgear on android , but it seems like there have some problems when importing vidgear's dependencies : ( first is mss I fix it with import mss; mss.mss(display = ":0") and another error comes out : x11 library not found. I tried to include python3-xlib in buildozer.spec requirement but seems like doesn't work : ( How can I fix it 😭😵 – Duck

um I'm using Windows OS and using the windows subsystem for linux to use buildozer to build apk , did you mean to edit the source code ? I tried try: from vidgear.gears import ScreenGear; ScreenGear() in my .py for testing but another error raised : ( unable to grab any instance on this system , are you running headless ? I can't find any solution online .... and would you mind to open a chat ? I really need help .... thanks – Duck

@PaulLam Let's Discuss here: gitter.im/vidgear/community – Audre

Answer 8

I tried all of the above but it did not give me the real-time screen update. You can try this. This code is tested and worked successfully and also give you a good fps output. You can also judge this by each loop time it's needed.

import numpy as np
import cv2
from PIL import ImageGrab as ig
import time

last_time = time.time()
while(True):
    screen = ig.grab(bbox=(50,50,800,640))
    print('Loop took {} seconds',format(time.time()-last_time))
    cv2.imshow("test", np.array(screen))
    last_time = time.time()
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

Diversified

Tried this on Windows with Python 3.6 and got around 10 fps with my i7 6700k CPU. – Maria

Loop took {} seconds. is that my FPS? – Tineid

No, its the time difference between two frame process... The less difference is very good. You can make another calculation like analyze the difference in every 60 seconds how many frames are processed... – Diversified

2.5 fps on a MacMini - while Markoe7's approach renders at > 65 fps – Karol

Answer 9

This is another method for using openCV record screen in full screen.

import cv2
import mss
import numpy
import time

with mss.mss() as sct:
    while True:
        last_time = time.time()
        
        img = numpy.array(sct.grab(sct.monitors[1]))
        cv2.imshow("OpenCV", img)
        
        print(f"fps: {1 / (time.time() - last_time)}")
        
        if cv2.waitKey(25) & 0xFF == ord("q"):
            cv2.destroyAllWindows()
            break

You can set the size of the recording space on the screen.

with mss.mss() as sct:
    monitor = {"top": 100, "left": 50, "width": 1280, "height": 800}
    while True:
        last_time = time.time()
        
        img = numpy.array(sct.grab(monitor))
        cv2.imshow("OpenCV", img)

        print(f"fps: {1 / (time.time() - last_time)}")
        
        if cv2.waitKey(25) & 0xFF == ord("q"):
            cv2.destroyAllWindows()
            break

Vastha

Answer 10

I've tried ImageGrab from PIL and it gave me 20fps which is ok but using win32 libraries gave me +40fps which is amazing!

I used this code by Frannecklp but it didn't work just fine so I needed to modify it:

-Firstly pip install pywin32 in case using the libraries

-import the libraries like this instead:

import cv2
import numpy as np
from win32 import win32gui
from pythonwin import win32ui
from win32.lib import win32con
from win32 import win32api

for geting a simple image screen do:

from grab_screen import grab_screen
import cv2
img = grab_screen()
cv2.imshow('frame',img)

and for getting frames:

while(True):
#frame = grab_screen((0,0,100,100))
frame = grab_screen()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q') or x>150:
    break

Insistence

Answer 11

This task is very simple with opencv, we are just capturing screenshots in loop, and converting it into frames. I created timer for screenrecording, in start you have to enter how many seconds you want to record:) Here is the code.

import cv2
import numpy as np
import pyautogui
from win32api import GetSystemMetrics
import time

#Take resolution from system automatically
w = GetSystemMetrics(0)
h =  GetSystemMetrics(1)
SCREEN_SIZE = (w,h)
fourcc = cv2.VideoWriter_fourcc(*"XVID")
out = cv2.VideoWriter("recording.mp4", fourcc, 20.0, (SCREEN_SIZE))
tim = time.time()
tp = int(input('How many times you want to record screen?->(Define value in Seconds): '))
tp = tp+tp
f = tim+tp
while True:
    img = pyautogui.screenshot()
    frame = np.array(img)
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    out.write(frame)
    tu = time.time()
    if tu>f:
        break
cv2.destroyAllWindows()
out.release()

So that's how you can use time in screen recording, you don't need to use imshow() because it shows infinitely our screen recording on-screen so output video looks weird.

Kunstlied

This works but it's really not fast, at least on linux where it depends on an external tool scrot to actually take the screenshots. It takes almost half a second with my 3 monitors, even if tell it to only screenshot a small region with pyautogui.screenshot(region=(...)). – Dashing

Answer 12

For those wondering how to achieve this on purely macOS, the following window capture code works great on the M1 Pro silicon MacBook Pro.

It captures your desired window by name and uses its ID to create a 'screenshot' using Quartz (python interface for interacting with the Core Graphics framework of macOS).

import numpy as np
import Quartz as QZ

class WindowCapture:

    # properties
    window_name = None
    window = None
    window_id = None
    window_width = 0
    window_height = 0

    # constructor
    def __init__(self, given_window_name=None):
        if given_window_name is not None:

            self.window_name = given_window_name
            self.window = self.get_window()

            if self.window is None:
                raise Exception('Unable to find window: {}'.format(given_window_name))

            self.window_id = self.get_window_id()

            self.window_width = self.get_window_width()
            self.window_height = self.get_window_height()

            self.window_x = self.get_window_pos_x()
            self.window_y = self.get_window_pos_y()
        else:
            raise Exception('No window name given')

    def get_window(self):
        windows = QZ.CGWindowListCopyWindowInfo(QZ.kCGWindowListOptionAll, QZ.kCGNullWindowID)
        for window in windows:
            name = window.get('kCGWindowName', 'Unknown')
            if name and self.window_name in name:
                
                return window
        return None
    
    def get_window_id(self):
        return self.window['kCGWindowNumber']

    def get_window_width(self):
        return int(self.window['kCGWindowBounds']['Width'])
    
    def get_window_height(self):
        return int(self.window['kCGWindowBounds']['Height'])

    def get_window_pos_x(self):
        return int(self.window['kCGWindowBounds']['X'])

    def get_window_pos_y(self):
        return int(self.window['kCGWindowBounds']['Y'])
    
    def get_image_from_window(self):
        core_graphics_image = QZ.CGWindowListCreateImage(
            QZ.CGRectNull,
            QZ.kCGWindowListOptionIncludingWindow,
            self.window_id,
            QZ.kCGWindowImageBoundsIgnoreFraming | QZ.kCGWindowImageNominalResolution
        )

        bytes_per_row = QZ.CGImageGetBytesPerRow(core_graphics_image)
        width = QZ.CGImageGetWidth(core_graphics_image)
        height = QZ.CGImageGetHeight(core_graphics_image)

        core_graphics_data_provider = QZ.CGImageGetDataProvider(core_graphics_image)
        core_graphics_data = QZ.CGDataProviderCopyData(core_graphics_data_provider)

        np_raw_data = np.frombuffer(core_graphics_data, dtype=np.uint8)

        numpy_data = np.lib.stride_tricks.as_strided(np_raw_data,
                                                shape=(height, width, 3),
                                                strides=(bytes_per_row, 4, 1),
                                                writeable=False)
        
        final_output = np.ascontiguousarray(numpy_data, dtype=np.uint8)

        return final_output

Here's an example of how to use it:

import cv2 as cv
from time import time
from windowcapture import WindowCapture

# initialise the WindowCapture class
wincap = WindowCapture('Name of my application window')

loop_time = time()

while(True):
    # get an updated image of the window you want
    screenshot = wincap.get_image_from_window()

    # show that image
    cv.imshow('Computer Vision', screenshot)

    print('FPS {:.2f}'.format(round(1 / (time() - loop_time), 2)))
    loop_time = time()

    # hold 'q' with the output window focused to exit.
    # waits 1 ms every loop to process key presses
    if cv.waitKey(1) == ord('q'):
        cv.destroyAllWindows()
        break

print('Done.')

Cornhusk