What sensors does ARCore use?

Question

What sensors does ARCore use: single camera, dual-camera, IMU, etc. in a compatible phone?

Also, is ARCore dynamic enough to still work if a sensor is not available by switching to a less accurate version of itself?

Acadian

It uses single camera and IMU. If you try an incompatible phone via rooting or sth. it will not be accurate because calibration can not be guaranteed. If that is what you are asking. – Iodic

Answer 1

Updated: May 10, 2022.

About ARCore and ARKit sensors

Google's ARCore, as well as Apple's ARKit, use a similar set of sensors to track a real-world environment. ARCore can use a single RGB camera along with IMU, what is a combination of an accelerometer, magnetometer and a gyroscope. Your phone runs world tracking at 60fps, while Inertial Measurement Unit operates at 1000Hz. Also, there is one more sensor that can be used in ARCore – iToF camera for scene reconstruction (Apple's name is LiDAR). ARCore 1.25 supports Raw Depth API and Full Depth API.

Read what Google says about it about COM method, built on Camera + IMU:

Concurrent Odometry and Mapping – An electronic device tracks its motion in an environment while building a three-dimensional visual representation of the environment that is used for fixing a drift in the tracked motion.

Here's Google US15595617 Patent: System and method for concurrent odometry and mapping.

• in 2014...2017 Google tended towards Multicam + DepthCam config (Tango project)
• in 2018...2020 Google tended to SingleCam + IMU config
• in 2021 Google returned to Multicam + DepthCam config

We all know that the biggest problem for Android devices is a calibration. iOS devices don't have this issue ('cause Apple controls its own hardware and software). A low quality of calibration leads to errors in 3D tracking, hence all your virtual 3D objects might "float" in a poorly-tracked scene. In case you use a phone without iToF sensor, there's no miraculous button against bad tracking (and you can't switch to a less accurate version of tracking). The only solution in such a situation is to re-track your scene from scratch. However, a quality of tracking is much higher when your device is equipped with ToF camera.

Here are four main rules for good tracking results (if you have no ToF camera):

1. Track your scene not too fast, not too slow

2. Track appropriate surfaces and objects

3. Use well lit environment when tracking

4. Don't track reflected of refracted objects

5. Horizontal planes are more reliable than vertical ones

SingleCam config vs MultiCam config

The one of the biggest problems of ARCore (that's ARKit problem too) is an Energy Impact. We understand that the higher frame rate is – the better tracking's results are. But the Energy Impact at 30 fps is HIGH and at 60 fps it's VERY HIGH. Such an energy impact will quickly drain your smartphone's battery (due to an enormous burden on CPU/GPU). So, just imagine that you use 2 cameras for ARCore – your phone must process 2 image sequences at 60 fps in parallel as well as process and store feature points and AR anchors, and at the same time, a phone must simultaneously render animated 3D graphics with Hi-Res textures at 60 fps. That's too much for your CPU/GPU. In such a case, a battery will be dead in 30 minutes and will be as hot as a boiler)). It seems users don't like it because this is not-good AR experience.

Algonquin

do you know if ARCore would use 2 cameras (stereo option) if the Android device had 2 cameras on board? – Acadian

I haven't tried it but I think it's possible to implement (but it's not as simple as it seems). – Algonquin

Motorola G6 is a compatible phone and Motorola G6 has 2 cameras (stereo) on the back. I was wondering whether ARCore switches to a mode where ARCore uses stereo vision for its computations in a compatible phone like Motorola G6. This is where I thought, maybe, ARCore has a more accurate mode with 2 cameras (stereo vision) and IMU versus the potentially less accurate mode with 1 camera and IMU. Any ideas? – Acadian

Why, using ARCore, does G6 have an advantage over Pixel 3, or Mate 20 Pro, or S9? Only because it has a stereo mode? Huawei Mate 20 Pro has three cameras but fulfils tracking using just one. – Algonquin

Tracking depends on RGB Data + IMU Data (in sync). The quality of tracking in ARCore doesn't get better if you'll use 2 cameras. Even if you have a Disparity Map from a stereo camera. In case you use 2 cameras (instead of one) along with IMU, it'll greatly increase a burden on CPU/GPU. Have you seen Energy Impact when you use ARCore? It's VERY HIGH at 60 fps. – Algonquin

It was a clarifying question. I wanted to know 1) whether one can choose to utilize one or more cameras with ARCore. 2) if so, whether anyone documented the accuracy of results with more sensors (2 cameras?, etc.) versus less sensors (1camera?, etc.) – Acadian

Also, for the record, I do not know if I am posing the right questions either. I just wanted to know the list sensors used and whether, for example, multiple cameras, and other sensors are opportunistically used by ARCore. – Acadian

I've already answered your question. Please, read it carefully))). 1. ARCore uses just one RGB camera at a time + IMU. 2. ARCore isn't switching to a less accurate version of itself when there's no any sensor. 3.You need't use several cameras because the quality of tracking depends on one camera + IMU. – Algonquin

How can you be sure that ARCore is using just one camera for trackin? Do you have any insight into implementation? If so please clarify, as from my observations majority of phones supported by ARCore have actually double camera in the back of the device. Would like to hear more about it, thanks. – Trangtranquada

As I told earlier, there's no necessity to use 2 or 3 cameras in ARCore, due to enormous energy impact. If you use just one Cam+IMU for augmented reality (plus rendering animated 3D graphics with particles at the same time) – your battery will be dead in 30 minutes. What will be if you use 2 or 3 cameras? Smartphone gets very hot and works only for 10-15 minutes. What's 15-minute AR-experience for? Nobody's interested in it. Nobody's interested in hot battery (it's a pity but it's only my logical reasoning, there's no official documentation on that topic). – Algonquin

Of course, there's an easy possibility for ARCore's developer (I mean Google) to implement 2 or 3 cameras (each running at 60 fps) for AR experience. But at the moment it doesn't make any sense. Battery's life is most important for user. – Algonquin

Second reason: Google thinks that a second lens is "unnecessary" for the Pixel 2 and Pixel 3 main "camera". So, what does it implement a second camera for (if ARCore greatly works with just one camera)? At the moment ARCore developers at Google have very difficult task: They need to optimise ARCore for 50 different non-calibrated smartphones that use just one rear camera. – Algonquin

@Trangtranquada and caansnews, I admit that my arguments might not be convincing. But Google produces single-camera Pixels and develops ARCore. There should be no advantage of a dual-camera phone over a single-camera phone. Don't you think so? Google has to sell their single-camera Pixels greatly running ARCore. To optimise ARCore for dual-camera phones is beyond Google's scope. Have a good day, guys! ))) – Algonquin

To tell you the truth, not only I do not know but also I do not have an opinion on what ARCore uses. The original question is either 1) to locate a documentation by ARCore which specifically addresses which subset of sensors or how ARCore uses; or 2) to locate a study and report by a third-party developer who checked all combinations and reported on the accuracy of the results. Like @kolboc, I am looking for a specific documented answer. Otherwise, I agree with all the insight you provided, and it makes sense, but it does not mean to say ARCore does this or that. – Acadian

Pity but Google doesn't provide any documentation on that topic! – Algonquin

In other words, smart phones have a suite of sensors these days but there is no way to tell what and how ARCore uses when the only input from ARCore is "ARCore uses camera + IMU." Camera: single or dual? IMU: software sensor which uses all the underlying hardware sensors or just the hardware sensor? (Android has software and hardware sensors and they mean different things. In general, software sensor is a fused sensor which uses the raw data from multiple sensors or computations.) – Acadian

@ARGeo, I agree that Google does not provide the specific answer. I was more hoping that one of us here spent some time to document the accuracy and figure it out, : ) – Acadian

ARCore is in its infancy now (like ARKit). Let's wait for couple of years at least. Google (and Apple too) gives us cool "instruments" within this API. – Algonquin

Sorry @caansnews, that's all I know about it)))) – Algonquin

It's 2023 now and still ARCore sucks with their motion tracking. Drift is there, no multi camera support and all devices are limited with 30 FPS. Apple AR kit is growing rapidly. – Sifuentes