Android 10 (api 29) camera2 api regression with wide-angle camera
Asked Answered
C

3

9

enter image description hereI'm using camera2 api in my camera app designed specifically for Google Pixel 3 XL. This device has two front facing cameras (wide-angle and normal). Thanks to multi-camera feature, I can access both physical camera devices simultaneously, and my app has a feature to toggle between those two cameras. Up until my recent upgrade to Android 10, I could accurately see two distinct results, but now my wide-angle capture frame has pretty much the same FOV (Field of View) as the normal camera one. So, the same code, same apk on Android 9 wide-angle capture result is wide, as expected, and after Andoird 10 upgrade - wide and normal cameras show practically identical FOV.

Here is a code snippet to demonstrate how I initialize both cameras and capture preview:

MainActivity.kt

 private val surfaceReadyCallback = object: SurfaceHolder.Callback {
        override fun surfaceChanged(p0: SurfaceHolder?, p1: Int, p2: Int, p3: Int) { }
        override fun surfaceDestroyed(p0: SurfaceHolder?) { }

        override fun surfaceCreated(p0: SurfaceHolder?) {

            // Get the two output targets from the activity / fragment
            val surface1 = surfaceView1.holder.surface  
            val surface2 = surfaceView2.holder.surface 

            val dualCamera = findShortLongCameraPair(cameraManager)!!
            val outputTargets = DualCameraOutputs(
                null, mutableListOf(surface1), mutableListOf(surface2))

            //Open the logical camera, configure the outputs and create a session
            createDualCameraSession(cameraManager, dualCamera, targets = outputTargets) { session ->

                val requestTemplate = CameraDevice.TEMPLATE_PREVIEW
                val captureRequest = session.device.createCaptureRequest(requestTemplate).apply {
                    arrayOf(surface1, surface2).forEach { addTarget(it) }
                }.build()

                session.setRepeatingRequest(captureRequest, null, null)
            }
        }
    }


    fun openDualCamera(cameraManager: CameraManager,
                       dualCamera: DualCamera,
                       executor: Executor = SERIAL_EXECUTOR,
                       callback: (CameraDevice) -> Unit) {

        cameraManager.openCamera(
            dualCamera.logicalId, executor, object : CameraDevice.StateCallback() {
                override fun onOpened(device: CameraDevice) { callback(device) }

                override fun onError(device: CameraDevice, error: Int) = onDisconnected(device)
                override fun onDisconnected(device: CameraDevice) = device.close()
            })
    }

    fun createDualCameraSession(cameraManager: CameraManager,
                                dualCamera: DualCamera,
                                targets: DualCameraOutputs,
                                executor: Executor = SERIAL_EXECUTOR,
                                callback: (CameraCaptureSession) -> Unit) {

        // Create 3 sets of output configurations: one for the logical camera, and
        // one for each of the physical cameras.
        val outputConfigsLogical = targets.first?.map { OutputConfiguration(it) }
        val outputConfigsPhysical1 = targets.second?.map {
            OutputConfiguration(it).apply { setPhysicalCameraId(dualCamera.physicalId1) } }
        val outputConfigsPhysical2 = targets.third?.map {
            OutputConfiguration(it).apply { setPhysicalCameraId(dualCamera.physicalId2) } }

        val outputConfigsAll = arrayOf(
            outputConfigsLogical, outputConfigsPhysical1, outputConfigsPhysical2)
            .filterNotNull().flatten()

        val sessionConfiguration = SessionConfiguration(SessionConfiguration.SESSION_REGULAR,
            outputConfigsAll, executor, object : CameraCaptureSession.StateCallback() {
                override fun onConfigured(session: CameraCaptureSession) = callback(session)
                override fun onConfigureFailed(session: CameraCaptureSession) = session.device.close()
            })


        openDualCamera(cameraManager, dualCamera, executor = executor) {
           it.createCaptureSession(sessionConfiguration)
        }
    }

DualCamera.kt Helper Class

data class DualCamera(val logicalId: String, val physicalId1: String, val physicalId2: String)

fun findDualCameras(manager: CameraManager, facing: Int? = null): Array<DualCamera> {
    val dualCameras = ArrayList<DualCamera>()

    manager.cameraIdList.map {
        Pair(manager.getCameraCharacteristics(it), it)
    }.filter {
        facing == null || it.first.get(CameraCharacteristics.LENS_FACING) == facing
    }.filter {
        it.first.get(CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES)!!.contains(
            CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA)
    }.forEach {
        val physicalCameras = it.first.physicalCameraIds.toTypedArray()
        for (idx1 in 0 until physicalCameras.size) {
            for (idx2 in (idx1 + 1) until physicalCameras.size) {
                dualCameras.add(DualCamera(
                    it.second, physicalCameras[idx1], physicalCameras[idx2]))
            }
        }
    }

    return dualCameras.toTypedArray()
}

fun findShortLongCameraPair(manager: CameraManager, facing: Int? = null): DualCamera? {

    return findDualCameras(manager, facing).map {
        val characteristics1 = manager.getCameraCharacteristics(it.physicalId1)
        val characteristics2 = manager.getCameraCharacteristics(it.physicalId2)

        val focalLengths1 = characteristics1.get(
            CameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS) ?: floatArrayOf(0F)
        val focalLengths2 = characteristics2.get(
            CameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS) ?: floatArrayOf(0F)

        val focalLengthsDiff1 = focalLengths2.max()!! - focalLengths1.min()!!
        val focalLengthsDiff2 = focalLengths1.max()!! - focalLengths2.min()!!

        if (focalLengthsDiff1 < focalLengthsDiff2) {
            Pair(DualCamera(it.logicalId, it.physicalId1, it.physicalId2), focalLengthsDiff1)
        } else {
            Pair(DualCamera(it.logicalId, it.physicalId2, it.physicalId1), focalLengthsDiff2)
        }

        // Return only the pair with the largest difference, or null if no pairs are found
    }.sortedBy { it.second }.reversed().lastOrNull()?.first
}

And you can see the result on the attached screenshot, the top left corner one has much wider FOV than the same camera but running on Android 10

Is this a known regression with Android 10? Has anyone noticed similar behavior?

Counterfeit answered 19/1, 2020 at 2:18 Comment(5)
Can you be more specific on how you're accessing the wide-angle camera and how you're setting the crop regions on it and the telephoto camera? As Android has improved its multi-camera APIs, the configuration of cameras on some Pixels has been updated to work better with the newer APIs. That said, it should still be completely possible to get the output you're looking for.Nonreturnable
@EddyTalvala I'm accessing the wide-angle (and normal) camera as follows: 1) Get physical camera ID form the list 2) set physical camera ID to outputConfiguration 3) get activeArraySize from camera characteristics and set SCALAR_CROP_REGION to capture request 4) create sessionConfiguration and add outputConfiguration to it 5) create capture session using sessionConfigurationCounterfeit
Were you able to solve this?Intercut
@LevonShirakyan did you find a solution?Glyconeogenesis
@LevonShirakyan I'm trying to implement the concept in xamarin android and having little bit trouble in understanding, do we have to open two different cameras in two different surface? like wide angle in surface1 and other camera in surface2?Alonsoalonzo
C
2

My understanding: I came across the same problem on my Pixel 3. It seems that the wide angle camera's frame has been cropped in the HAL layer before combination. Actually the FOV is not totally the same, as there is a little disparity between left and right camera. However, the default zoom level of wide camera seems to change according to the focal length.

But I could not find any official documentation about it. In Android 10, it claims improved the fusing of physical cameras: https://developer.android.com/about/versions/10/features#multi-camera

1

Solution:

If you wish to access the raw data from the wide angle front camera, you can create 2 camera sessions for both physical cameras instead of a single session for the logical camera.

Updated:

You can use the setPhysicalCameraKey to reset the zoom level https://developer.android.com/reference/android/hardware/camera2/CaptureRequest.Builder#setPhysicalCameraKey(android.hardware.camera2.CaptureRequest.Key%3CT%3E,%20T,%20java.lang.String)

Cathodoluminescence answered 7/5, 2020 at 8:11 Comment(1)
camChars.availablePhysicalCameraRequestKeys returns null so I cannot use setPhysicalCameraKey, only camChars.physicalCameraIds returns 3 idsGlyconeogenesis
M
1

The regression you observed is a behavior change on Pixel 3/Pixel 3XL between Android 9 and Android 10. It is not an Android API change per se, but something the API allows the devices change behavior on; other devices may be different.

The camera API allows the physical camera streams to be cropped to match the field-of-view of the logical camera stream.

Mcphail answered 11/3, 2021 at 0:39 Comment(0)
A
1

On pixel 3 (Android 11), probing cameras using CameraManager.getCameraIdList() returns 4 IDs: 0, 1, 2, 3

  • 0: Back Camera : Physical Camera Stream
  • 1: Front camera : Logical camera with two physical camera ID's
  • 2: Front camera normal: Physical Camera Stream
  • 3: Front camera widelens: Physical Camera Stream

As user DannyLin suggested, opening 2 physical camera streams (2,3) seems to do the job. Note that other combinations such as (0, 1), (1, 2) etc do not work (only the first call to openCamera() goes through and the second call fails). Here's a snapshot of the physical camera streams for two front camera's.

enter image description here

Astilbe answered 28/4, 2021 at 15:8 Comment(4)
So do we need to call CameraManager.openCamera(0) and then call setPhysicalCameraId(2) and setPhysicalCameraId(3) for OutputConfigurations or what? also I want access just wide lens camera, so I don't want to have two surfaceview, just oneBreechloader
@Breechloader Here is an implementation for your reference (the relevant code for second camera is commented out though). And Yes you can use just one surfaceView (either or both wide and non-wide camera).Astilbe
but this sample just tries to open cameras with CameraManager.openCamera() and setPhysicalCameraId, physical cameras should NOT be opened with CameraManager.openCamera() and there is no method to check that it will work for such cameras, only logical camera ids are guaranteed to work with itBreechloader
@Breechloader As I see the linked sample uses Logical Camera ID and works. You can ignore the commented code in the above sample.Astilbe

© 2022 - 2024 — McMap. All rights reserved.