Using the onFrameAvailable() in Jacobi Google Tango API

Question

Question: Does anyone know how to get the Tango's color camera image buffer using the Tango Java (Jacobi) API onFrameAvailable() callback?

Background:

I have an augmented reality application that displays video in the background of the Tango. I've successfully created the video overlay example using the the Java API (Jacobi) following this example. My application works fine, and the video is rendered in the background properly.

As part of the application, I'd like to store a copy of the video backbuffer when the user presses a button. Therefore, I need access to the camera's RGB data.

According to the Jacobi release notes, any class desiring access to the camera RGB data should implement the new onFrameAvailable() method in the OnTangoUpdateListener. I did this, but I don't see any handle or arguments to actually get the pixels:

Java API

@Override
public void onFrameAvailable(int cameraId) {
    //Log.w(TAG, "Frame available!");
    if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {
        tangoCameraPreview.onFrameAvailable();
    }
}

as shown, onFrameAvailable only has one argument, and integer designating the id of the camera generating the view. Contrast this with the C-library call back, which provides access to the image buffer:

C API

TangoErrorType TangoService_connectOnFrameAvailable(
    TangoCameraId id, void* context,
    void (*onFrameAvailable)(void* context, TangoCameraId id,
                             const TangoImageBuffer* buffer));

I was expecting the Java method to have something similar to the buffer object in the C API call.

What I've Tried

I tried extending the TangoCameraPreview class and saving the image there, but I only get a black background.

public class CameraSurfaceView extends TangoCameraPreview {


    private boolean takeSnapShot = false;

    public void takeSnapShot() {
        takeSnapShot = true;
    }

    /**
     * Grabs a copy of the surface (which is rendering the Tango color camera)
     * https://mcmap.net/q/1229307/-how-to-take-a-screenshot-of-android-39-s-surface-view
     */
    public void screenGrab2(){

        int width = this.getWidth();
        int height = this.getHeight();
        long fileprefix = System.currentTimeMillis();

        View v= getRootView();
         v.setDrawingCacheEnabled(true);
        // this is the important code :)  
        // Without it the view will have a dimension of 0,0 and the bitmap will be null          
        v.measure(MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED),
        MeasureSpec.makeMeasureSpec(0, MeasureSpec.UNSPECIFIED));
        v.layout(0, 0, width, height);

        v.buildDrawingCache(true);

        Bitmap image = v.getDrawingCache();

        //TODO:  make seperate subdirctories for each exploitation sessions
        String targetPath =Environment.getExternalStorageDirectory()  + "/RavenEye/Photos/";
        String imageFileName = fileprefix + ".jpg";   

        if(!(new File(targetPath)).exists()) {
            new File(targetPath).mkdirs();
        }

        try {           
            File targetDirectory = new File(targetPath);
            File photo=new File(targetDirectory, imageFileName);
            FileOutputStream fos=new FileOutputStream(photo.getPath());
            image.compress(CompressFormat.JPEG, 100, fos);          
            fos.flush();
            fos.close();
            Log.i(this.getClass().getCanonicalName(), "Grabbed an image in target path:" + targetPath);
        } catch (FileNotFoundException e) {
            Log.e(CameraPreview.class.getName(),"Exception " + e);
            e.printStackTrace();
        } catch (IOException e) {
            Log.e(CameraPreview.class.getName(),"Exception " + e);
            e.printStackTrace();
        }   

    }


    /**
     * Grabs a copy of the surface (which is rendering the Tango color camera)
     */
    public void screenGrab(){

        int width = this.getWidth();
        int height = this.getHeight();
        long fileprefix = System.currentTimeMillis();

        Bitmap image = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);

        Canvas canvas = new Canvas(image);
        canvas.drawBitmap(image, 0, 0, null);

        //TODO:  make seperate subdirctories for each exploitation sessions
        String targetPath =Environment.getExternalStorageDirectory()  + "/RavenEye/Photos/";
        String imageFileName = fileprefix + ".jpg";   

        if(!(new File(targetPath)).exists()) {
            new File(targetPath).mkdirs();
        }

        try {           
            File targetDirectory = new File(targetPath);
            File photo=new File(targetDirectory, imageFileName);
            FileOutputStream fos=new FileOutputStream(photo.getPath());
            image.compress(CompressFormat.JPEG, 100, fos);          
            fos.flush();
            fos.close();
            Log.i(this.getClass().getCanonicalName(), "Grabbed an image in target path:" + targetPath);
        } catch (FileNotFoundException e) {
            Log.e(CameraPreview.class.getName(),"Exception " + e);
            e.printStackTrace();
        } catch (IOException e) {
            Log.e(CameraPreview.class.getName(),"Exception " + e);
            e.printStackTrace();
        }   

    }

    @Override
    public void onFrameAvailable() {
        super.onFrameAvailable();
        if(takeSnapShot) {
            screenGrab();
            takeSnapShot = false;

        }
    }

    public CameraSurfaceView(Context context) {
        super(context);
        // TODO Auto-generated constructor stub
    }
}

Where I'm Heading

I'm preparing to root the device, and then using the onFrameAvailable method to cue an external root process such as one of these:

post 23610900

post 10965409

post 4998527

I'm hoping I can find a way to avoid the root hack.

Thank you in advance!

Doorframe

Answer 1

OK, I figured out a way to make it work.

Update: My working solution is here:

https://github.com/stevehenderson/GoogleTango_AR_VideoCapture

I essentially set up a "man (renderer) in the middle" attack on the rendering pipeline. This approach intercepts the SetRenderer call from the TangoCameraPreview base class, and allows one to get access to the base renderer's OnDraw() method and the GL context. I then add additional methods to this extended renderer that allow reading of the GL buffer.

General approach

1) Extend the TangoCameraPreview class (e.g. in my example ReadableTangoCameraPreview). Override the setRenderer(GLSurfaceView.Renderer renderer), keeping a reference to the base renderer, and replacing the renderer with your own "wrapped" GLSUrface.Renderer renderer that will add methods to render the backbuffer to an image on the device.

2) Create your own GLSurfaceView.Renderer Interface (e.g. my ScreenGrabRenderer class ) that implements all the GLSurfaceView.Renderer methods, passing them on to the base renderer captured in Step 1. Also, add a few new methods to "cue" when you want to grab the image.

3) Implement the ScreenGrabRenderer described in step 2 above.

4) Use a callback interface (my TangoCameraScreengrabCallback) to communicate when an image has been copied

It works pretty well, and allows one to grab the camera bits in an image without rooting the device.

Note: I haven't had the need to closely synchronize my captured images with the point cloud. So I haven't checked the latency. For best results, you may need to invoke the C methods proposed by Mark.

Here's what each of my classes looks like..

///Main Activity Class where bulk of Tango code is
.
.
.

// Create our Preview view and set it as the content of our activity.
mTangoCameraPreview = new ReadableTangoCameraPreview(getActivity());

RelativeLayout preview = (RelativeLayout) view.findViewById(R.id.camera_preview);
preview.addView(mTangoCameraPreview);


.
.
.
//When you want to take a snapshot, call the takeSnapShotMethod()
//(you can make this respond to a button)
mTangoCameraPreview.takeSnapShot();
.
.
.
.
.
//Main Tango Listeners
@Override
public void onFrameAvailable(final int cameraId) {
    // Update the UI with TangoPose information
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {
                tangoCameraPreview.onFrameAvailable();
            }       
        }
    });
}

---

ReadableTangoCameraPreview Class

public class ReadableTangoCameraPreview extends TangoCameraPreview implements  TangoCameraScreengrabCallback  {

    Activity mainActivity;
    private static final String TAG = ReadableTangoCameraPreview.class.getSimpleName();

    //An intercept renderer
    ScreenGrabRenderer screenGrabRenderer;

    private boolean takeSnapShot = false;

    @Override
    public void setRenderer(GLSurfaceView.Renderer renderer) {  
        //Create our "man in the middle"
        screenGrabRenderer= new ScreenGrabRenderer(renderer);

        //Set it's call back
        screenGrabRenderer.setTangoCameraScreengrabCallback(this);

        //Tell the TangoCameraPreview class to use this intermediate renderer
        super.setRenderer(screenGrabRenderer);
        Log.i(TAG,"Intercepted the renderer!!!");       
    }


    /**
     * Set a trigger for snapshot.  Call this from main activity
     * in response to a use input
     */
    public void takeSnapShot() {
        takeSnapShot = true;
    }   

    @Override
    public void onFrameAvailable() {
        super.onFrameAvailable();
        if(takeSnapShot) {
            //screenGrabWithRoot();
            screenGrabRenderer.grabNextScreen(0,0,this.getWidth(),this.getHeight());
            takeSnapShot = false;           
        }
    }

    public ReadableTangoCameraPreview(Activity context) {
        super(context); 
        mainActivity = context;     

    }

    public void newPhoto(String aNewPhotoPath) {
        //This gets called when a new photo was  grabbed created in the renderer
        Log.i(TAG,"New image available at" + aNewPhotoPath);    
    }

}

---

ScreenGrabRenderer Interface

(Overloads the TangoCameraPreview default Renderer)

/**
 * This is an intermediate class that intercepts all calls to the TangoCameraPreview's
 * default renderer.
 * 
 * It simply passes all render calls through to the default renderer.
 * 
 * When required, it can also use the renderer methods to dump a copy of the frame to a bitmap
 * 
 * @author henderso
 *
 */
public class ScreenGrabRenderer implements GLSurfaceView.Renderer  {


    TangoCameraScreengrabCallback mTangoCameraScreengrabCallback;

    GLSurfaceView.Renderer tangoCameraRenderer;
    private static final String TAG = ScreenGrabRenderer.class.getSimpleName();

    private String lastFileName = "unset";

    boolean grabNextScreen = false;

    int grabX = 0;
    int grabY = 0;
    int grabWidth = 640;
    int grabHeight = 320;

    public void setTangoCameraScreengrabCallback(TangoCameraScreengrabCallback aTangoCameraScreengrabCallback) {
        mTangoCameraScreengrabCallback = aTangoCameraScreengrabCallback;
    }

    /**
     * Cue the renderer to grab the next screen.  This is a signal that will
     * be detected inside the onDrawFrame() method
     * 
     * @param b
     */
    public void grabNextScreen(int x, int y, int w, int h) {
        grabNextScreen = true;
        grabX=x;
        grabY=y;
        grabWidth=w;
        grabHeight=h;
    }

    @Override
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {
        tangoCameraRenderer.onSurfaceCreated(gl, config);

    }

    @Override
    public void onSurfaceChanged(GL10 gl, int width, int height) {
        tangoCameraRenderer.onSurfaceChanged(gl, width, height);        
    }

    @Override
    public void onDrawFrame(GL10 gl) {
        tangoCameraRenderer.onDrawFrame(gl);    
        if(grabNextScreen) {
            screenGrab(gl);
            grabNextScreen=false;
        }
    }


    /**
     * 
     * Creates a bitmap given a certain dimension and an OpenGL context
     *  
     * This code was lifted from here:
     * 
     * https://mcmap.net/q/57097/-capture-screen-of-glsurfaceview-to-bitmap 
     */
    private Bitmap createBitmapFromGLSurface(int x, int y, int w, int h, GL10 gl)
            throws OutOfMemoryError {
        int bitmapBuffer[] = new int[w * h];
        int bitmapSource[] = new int[w * h];
        IntBuffer intBuffer = IntBuffer.wrap(bitmapBuffer);
        intBuffer.position(0);

        try {
            gl.glReadPixels(x, y, w, h, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, intBuffer);
            int offset1, offset2;
            for (int i = 0; i < h; i++) {
                offset1 = i * w;
                offset2 = (h - i - 1) * w;
                for (int j = 0; j < w; j++) {
                    int texturePixel = bitmapBuffer[offset1 + j];
                    int blue = (texturePixel >> 16) & 0xff;
                    int red = (texturePixel << 16) & 0x00ff0000;
                    int pixel = (texturePixel & 0xff00ff00) | red | blue;
                    bitmapSource[offset2 + j] = pixel;
                }
            }
        } catch (GLException e) {
            Log.e(TAG,e.toString());
            return null;
        }

        return Bitmap.createBitmap(bitmapSource, w, h, Bitmap.Config.ARGB_8888);
    }


    /**
     * Writes a copy of the GLSurface backbuffer to storage
     */
    private void screenGrab(GL10 gl) {
        long fileprefix = System.currentTimeMillis();
        String targetPath =Environment.getExternalStorageDirectory()  + "/RavenEye/Photos/";
        String imageFileName = fileprefix + ".png";   
        String fullPath = "error";

        Bitmap image = createBitmapFromGLSurface(grabX,grabY,grabWidth,grabHeight,gl);
        if(!(new File(targetPath)).exists()) {
            new File(targetPath).mkdirs();
        }
        try {           
            File targetDirectory = new File(targetPath);
            File photo=new File(targetDirectory, imageFileName);
            FileOutputStream fos=new FileOutputStream(photo.getPath());
            image.compress(CompressFormat.PNG, 100, fos);          
            fos.flush();
            fos.close();
            fullPath =targetPath + imageFileName;
            Log.i(TAG, "Grabbed an image in target path:" + fullPath);      

            ///Notify the outer class(es)
            if(mTangoCameraScreengrabCallback != null) {
                mTangoCameraScreengrabCallback.newPhoto(fullPath);
            } else {
                Log.i(TAG, "Callback not set properly..");
            }

        } catch (FileNotFoundException e) {
            Log.e(TAG,"Exception " + e);
            e.printStackTrace();
        } catch (IOException e) {
            Log.e(TAG,"Exception " + e);
            e.printStackTrace();
        }   
        lastFileName = fullPath;
    }


    /**
     * Constructor
     * @param baseRenderer
     */
    public ScreenGrabRenderer(GLSurfaceView.Renderer baseRenderer) {
        tangoCameraRenderer = baseRenderer;     
    }
}

---

TangoCameraScreengrabCallback Interface (not required unless you want to pass info back from the screen grab renderer)

    /*
     * The TangoCameraScreengrabCallback is a generic interface that provides callback mechanism 
     * to an implementing activity.
     * 
     */
    interface TangoCameraScreengrabCallback {
        public void newPhoto(String aNewPhotoPath);
    }

Doorframe

Awesome - the Java folks are gonna love this, you gave them what Google didn't! – Venous

Yeah it's pretty slow right now, and sometimes hangs the Tango thread when I copy the bits. I'm going to play with it some more and see if I can't post some demo code... – Doorframe

Answer 2

I haven't tried on the latest release, but it was the absence of this functionality that drove me to the C API where I could get image data - a recent post, I think on the G+ page, seemed to indicate that the Unity API now returns image data as well - for a company that wants to keep scolding us when we don't use Java, it certainly is an odd lag :-)

Venous