How can I access my laptop's built-in infrared webcam using Python?

Question

I'm trying to access my laptop's built-in infrared webcam (intended for windows hello) in a Python project. I can access the normal RGB camera quite easily using the VideoCapture class from OpenCV, however can't find a way to access the infrared camera. I tried passing different indices to the VideoCapture class, however only "0" works, which is the normal webcam.

There are ways to do this using the Windows API, however I can't find a way to access this in Python.

Some people have been able to access the windows hello camera on Ubuntu using fswebcam.

Is there anyway to access the infrared camera using python? Maybe some way of interfacing with the Windows API using pywin32? Any help getting started with this would be appreciated. Alternatively, is there anything similar to fswebcam for Windows that would let me do the same?

Launcher

Have you tried libuvc for linux? I have used this on Android and been able to access external web cams. – Wenwenceslaus

Thanks for the suggestion, but unfortunately that wouldn't work because I have to use windows for this project – Launcher

did you try vlc first to check if your device is listed ? You probably also have to enable the camnera first. – Roydd

Answer 1

I'm a bit late but maybe I can help others. You can access the IR camera by WinRT. If you want to do it with Python, a maintained winrt alternative winsdk should be used.

I followed this tutorial: Process media frames with MediaFrameReader

I skipped and simplified some sections of the code, but it does the job. Captured frames are shown in an OpenCV window. You can quit by pressing 'q'.

You have to pay attention for two constants in the script. camera_image_width and camera_image_height define the resolution of the captured frames. For my hardware they are 640 and 360. I don't know if this is a standard resolution for previewing a camera, they may be different for other hardwares. You can determine them by calling software_bitmap.pixel_height and software_bitmap.pixel_height after line 88.

The following Python packages must be installed:

• asyncio
• numpy
• opencv-python
• winsdk

Here's the whole script:

import asyncio
import time
import numpy as np

import cv2

from winsdk.windows.media.capture import MediaStreamType
from winsdk.windows.media.capture import MediaCapture
from winsdk.windows.media.capture import MediaCaptureInitializationSettings
from winsdk.windows.media.capture import MediaCaptureSharingMode
from winsdk.windows.media.capture import MediaCaptureMemoryPreference
from winsdk.windows.media.capture import StreamingCaptureMode

from winsdk.windows.media.capture.frames import MediaFrameSourceGroup
from winsdk.windows.media.capture.frames import MediaFrameSourceInfo
from winsdk.windows.media.capture.frames import MediaFrameSourceKind
from winsdk.windows.media.capture.frames import MediaFrameReader

from winsdk.windows.graphics.imaging import SoftwareBitmap
from winsdk.windows.graphics.imaging import BitmapPixelFormat

from winsdk.windows.storage.streams import Buffer

camera_image_width = 640
camera_image_height = 360

async def main():
    #####  Find the first infra source group  #####

    media_frame_source_groups = await MediaFrameSourceGroup.find_all_async()

    infra_source_group: MediaFrameSourceGroup
    infra_source_info: MediaFrameSourceInfo

    # Iterating through source groups
    for source_group in media_frame_source_groups:
        for source_info in source_group.source_infos:
            if source_info.media_stream_type == MediaStreamType.VIDEO_PREVIEW and source_info.source_kind == MediaFrameSourceKind.INFRARED:
                infra_source_group = source_group
                infra_source_info = source_info

    # Check if there's available infra source
    if infra_source_group is None or infra_source_info is None:
        print("No infra source was found!")
        return

    #####  Getting frames from infra camera  #####

    with MediaCapture() as media_capture:
        # Settings for MediaCapture object
        media_capture_settings = MediaCaptureInitializationSettings()
        media_capture_settings.source_group = infra_source_group
        media_capture_settings.sharing_mode = MediaCaptureSharingMode.EXCLUSIVE_CONTROL
        media_capture_settings.memory_preference = MediaCaptureMemoryPreference.CPU
        media_capture_settings.streaming_capture_mode = StreamingCaptureMode.VIDEO
        await media_capture.initialize_async(media_capture_settings)

        # Init frame reader
        media_frame_reader: MediaFrameReader = await media_capture.create_frame_reader_async(media_capture.frame_sources[infra_source_info.id])
        await media_frame_reader.start_async()

        # Each frame must be dumped into a buffer
        frame_buffer = Buffer(camera_image_width * camera_image_height * 4)

        # Wait for frames
        while True:
            # Capturing with 20fps
            time.sleep(0.05)

            # Get the latest available frame
            media_frame_reference = media_frame_reader.try_acquire_latest_frame()

            # If the frame is null, skip this cycle
            if media_frame_reference is None:
                continue

            # Try to read frame
            with media_frame_reference:
                # Boilerplate bullshit to extract bitmap data
                video_media_frame = media_frame_reference.video_media_frame

                if video_media_frame is None:
                    return
                
                software_bitmap = video_media_frame.software_bitmap

                if software_bitmap is None:
                    return
                
                #####  The bitmap is just fine  #####

                with software_bitmap:
                    # Convert image format
                    grayscale_bitmap = SoftwareBitmap.convert(software_bitmap, BitmapPixelFormat.RGBA8)

                    # Dump image data to byte buffer
                    grayscale_bitmap.copy_to_buffer(frame_buffer)
                    
                    # Create actual OpenCV image from byte buffer
                    image_array = np.frombuffer(frame_buffer, dtype=np.uint8).reshape(camera_image_height, camera_image_width, 4)

                    # If the image is all black, skip showing
                    if cv2.countNonZero(cv2.cvtColor(image_array, cv2.COLOR_RGBA2GRAY)) == 0:
                        continue
                    
                    cv2.imshow('Infrared camera', image_array)
                    
                    if cv2.waitKey(1) == ord('q'):
                        break

        cv2.destroyAllWindows()

asyncio.run(main())

Good luck!

Gomer

Answer 2

That example you provided, doesn't seem to use the win32 api, but rather the dotnet framework, specifically the Windows.Media.Capture.Frames.MediaFrameSourceGroup class.

It looks like you'll be able to access the dotnet framework using http://pythonnet.github.io/ . After that it should be a case of porting that C# code over to Python.

Frug

Answer 3

You need to access WinRT API, not Win32 API.

Use this library: https://github.com/Microsoft/xlang/tree/master/src/package/pywinrt/projection and you should be able to call invoke necessary parts in WinRT from the code you posted.

Dorfman