How to take high-res picture while sensing depth using project tango

Question

How take picture using project tango ?

I read this answer: Using the onFrameAvailable() in Jacobi Google Tango API

which works for grabbing a frame but picture quality is not great. Is there any takePicture equivalent ?

Note that java API

        public void onFrameAvailable(int cameraId) {
            if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {
                mTangoCameraPreview.onFrameAvailable();
            }
        }

does not provide rgb data. If I use android camera to take picture, tango can not sense depth. There I will have to use TangoCameraPreview.

Thanks

Infer

Answer 1

You don't have to use TangoCameraPreview to get frames in Java. It is really just a convenience class provided to help with getting video on the screen. It appears to be implemented entirely in Java with calls to com.google.atap.tangoservice.Tango (i.e no calls to unpublished APIs). In fact, if you look inside the Tango SDK jar file, you can see that someone accidentally included a version of the source file - it has some diff annotations and may not be up to date but examining it is still instructive.

I prefer not to use TangoCameraPreview and instead call Tango.connectTextureId() and Tango.updateTexture() myself to load frame pixels into an OpenGL texture that I can then use however I want. That is exactly what TangoCameraPreview does under the hood.

The best way to capture a frame in pure Java is to draw the texture at its exact size (1280x720) to an offscreen buffer and read it back. This also has the side effect of converting the texture from whatever YUV format it has into RGB (which may or may not be desirable). In OpenGL ES you do this using a framebuffer and renderbuffer.

Adding the framebuffer/renderbuffer stuff to a program that can already render to the screen isn't a lot of code - about on par with the amount needed to save a file - but it is tricky to get right when you do it for the first time. I created an Android Studio sample capture app that saves a Tango texture as a PNG to the pictures folder (when you tap the screen) in case that is helpful for anyone.

Overtly

I skimmed your code and it seems to be similar to answer listed in my question. Both would work but they won't result into sharp high-res image. Capturing display buffer before rendering and saving into a file is just a hack to capture video frames. I am looking for a method similar to takePicture() in android so that whenever I press a button -- I have full HD image, auto-focussed, along with depth map and camera pose. I can do post processing to extract various things from image such as features, colors, pixel to depth mapping, etc etc in future. – Infer

I don't think that's possible without disconnecting Tango when you need to take your shot, and even if that works it will have high latency. – Overtly

Upvoted because you snooped the jar and found the source - embarrassed I missed that one myself :-( Thank you for a great catch! That said, the docs seem to strongly indicate grabbing the camera means that tango can't do point clouds (pose is from fisheye as I understand it) – Conservationist