Hololens - Access Camera Frames

Question

How can I access all frames in a video stream from the hololens camera, when in 3D mode? I'm using C#.

The unity VideoCapture class doesn't seem to provide this: http://docs.unity3d.com/550/Documentation/ScriptReference/VR.WSA.WebCam.VideoCapture.html

and the microsoft MediaCapture class with method MediaCapture.StartPreviewToCustomSinkAsync is not very well documented. https://msdn.microsoft.com/en-us/library/windows/desktop/ms701626(v=vs.85).aspx

Fornicate

Answer 1

You can try using the MediaCapture FrameReader API

Documentation: https://msdn.microsoft.com/en-gb/windows/uwp/audio-video-camera/process-media-frames-with-mediaframereader#create-a-frame-reader-for-the-frame-source

Sample code: https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraFrames

Jerky

@Fornicate does it work with hololens and if yes can you please guide me with sample code? – Baudelaire

Answer 2

We made an open source project called CameraStream to address this need. As a Unity plugin, it uses MediaCapture to feed the byte array into Unity (along with locatable matrices). From there you can assign the bytes to a Texture2D, as shown in the provided Unity example.

Our VideoCapture class mimic's Unity's class of the same name so that developers would be familiar with how to use it. We're also assuming that this functionality will be provided by Unity out-of-the-box eventually.

Valleau

You're the MVP I was looking for. I can't believe I came across that answer right when it was posted... Thanks a lot mate. – Lajuanalake

@Eric Smith: Now we have 2020, do you know if there is some way of getting the raw video stream pixels? – Frighten

@Eric Smith: Does your open source project also provide a way of getting the Video + the rendered 3D content? I would like to be able to stream a video with what the Hololens user sees. – Frighten