ARCore – Raw Depth Data from rear Android depth camera

Question

I can't figure out how can I get raw depth data from my Xiaomi Redmi Note 8 Pro with rear ToF depth sensor.

I've tried:

• Camera2Format from android samples. No results (and I'm not alone)
• get physicalId from rear camera as described in android developers blog. No result.
• ARCore API can't handle depth.

I saw several similar questions (one, two) without answers. Is it so difficult to capture raw depth from depth camera in Android?

Robb

If you look at the end of this discussion there is modified version of Camera2RawFragment which the author uses to access the DEPTH image data: groups.google.com/a/android.com/forum/#!topic/…. Hopefully, this helps although its worth reading the full thread on the accuracy of the data. – Noddle

Answer 1

Raw Depth API vs Full Depth API

At Google I/O 2021, the Raw Depth API for ARCore 1.24 version was announced in addition to the existing Full Depth API, working since ARCore 1.18. This second-Gen depth API gives you the ability to merge Raw Depth data coming from iToF sensor with data coming from Depth-from-Motion ML-algorithm. The resulted data can be retrieved through 16-bit buffer stream, where 13-bit is reserved for depth map and 3-bit is reserved for confidence map.

At Google I/O 2022, Full Depth API was updated for users running ARCore 1.31. Now, all 16 bits per pixel are used, allowing app to represent the depth within a distance of 65,535 millimeters.

Today ARCore developers can use Raw Depth API for Precise Measurements and for ops on Reconstructed Mesh. On the other hand, Full Depth API is ideally suitable for People/Objects Occlusion and Scene Relighting purposes.

The following table represents three important parameters:

	Full Depth API (v1.31+)	Raw Depth API (v1.24+)	Full Depth API (v1.18+)
Accuracy	Bad	Awesome	Bad
Coverage	All pixels	Not all pixels	All pixels
Distance	0 to 65.5 m	0.5 to 5.0 m	0 to 8.2 m

Raw Depth is available on all devices that support the Depth API. The Raw Depth API, like the Full Depth API, does not require a supported hardware depth sensor, such as a time-of-flight (iToF) sensor. However, both the Raw Depth API and the Full Depth API make use of any supported hardware sensors that a device may have.

The list of ARCore Depth API supported devices is growing. Pay attention to the last column called Comments - info about Depth API support is there.

Rawlinson