TangoService_connectOnFrameAvailable() gets stuck or crashes using Google Tango Leibniz Release 1.10

Question

There seems to be an issue concering the reception of color frames in the Leibniz Release 1.10 as well: when having registered a callback using TangoService_connectOnFrameAvailable(TANGO_CAMERA_COLOR,NULL,onFrameAvailable) the callback onFrameAvailable() will either never ever be called or TangoService_connectOnFrameAvailable() crashes with the following error:

04-20 13:29:44.384: E/tango_client_api(4712): TangoErrorType TangoService_connectOnFrameAvailable(TangoCameraId, void*, void ()(void, TangoCameraId, const TangoImageBuffer*)): Internal Error: connectSurface(), cam id 0, failed internally.

The release notes say

[...] config_enable_color_camera has been added to the configuration flags. We recommend you always explicitly set this flag to true if accessing the color camera. You must set the flag true for TangoService_connectOnFrameAvailable() or TangoService_connectTextureId() to succeed after TangoService_connect() is called. [...]

Thus, if I set that flag to true between the calls of TangoService_connect() and TangoService_connectOnFrameAvailable(), the callback onFrameAvailable() will never ever be called, if I set that flag to true before TangoService_connect() TangoService_connectOnFrameAvailable() will always crash.

Thus, what am I doing wrong? Is there a code snippet available or something? That would be really helpful... Unfortunately, none of the examples use color frames...

Man, after having had similar problems with the Kalman Release 1.9, I begin to wonder if the SDKs are thoroughly tested before being released in the first place...

Micronesia

OK, this release is starting to smell like yesterdays fish - all the people that have been keeping their heads down are popping up with issues :=( I have yet to integrate these changes, but I've read the same thing you have and I can only (weakly) suggest you look very closely at the callback binding, just to be very very sure. You are spot on with respect to the examples, a simple GitHub search proves that. As for your final point, I've fired people for exactly this behavior. – Jenness

Apologies, that you are experiencing problems. Is this still happening? I am asking this because, there was a bit of leeway in timing between when the TangoCore was updated on PlayStore and when the OTA went out (which can potentially cause this issue, if OTA and TangoCore are mismatched). I just want to make sure that you are are updated on both TangoCore and OTA before diagnosing it. Also, make sure you have permissions for camera in the android manifestl. – Endomorphic

Well, on my Tango tablet there actually was that mismatch between OTA and TangoCore you are speaking of... After having everything carefully updated once more I finally was able to bind a callback... Thanks so much... For those of you who are interested in how to convert the NV21 to an RGB frame I attach my code below... – Epicrisis

Answer 1

Alright assuming that the problem is not what I mentioned in the comments section. Here is code snippet testing the onFrameAvailable Callback.

Note: I have modified the HelloTangoJni Example from the Tango-examples-c repository for this.

In TangoHandler.h add

 TangoErrorType ConnectYUVFrameCallback();

Modify TangoHandler.cc

TangoErrorType TangoHandler::SetupConfig() {
  // TANGO_CONFIG_DEFAULT is enabling Motion Tracking and disabling Depth
  // Perception.
  tango_config_ = TangoService_getConfig(TANGO_CONFIG_DEFAULT);
  if (tango_config_ == nullptr) {
  return TANGO_ERROR;
  }
  TangoConfig_setBool(tango_config_,"config_enable_color_camera",true);
  return TANGO_SUCCESS;
}


TangoErrorType TangoHandler::ConnectYUVFrameCallback() {
    TangoErrorType onFrameErrorType=TangoService_connectOnFrameAvailable( TANGO_CAMERA_COLOR, NULL, onFrameAvailable);
    if( onFrameErrorType!= TANGO_SUCCESS)
    {
         LOGI("GOOGLE TANGO ONFRAMEAVAILABLE FAILED!");
    }
    LOGI("GOOGLE TANGO ONFRAMEAVAILABLE SUCCESS!");
    return onFrameErrorType;
}

static void onFrameAvailable( void* context, const TangoCameraId id, const TangoImageBuffer* buffer )
{
  int width = buffer->width;
  int height = buffer->height;
  LOGI("width and height is: %d,%d",width,height);
}

In TangoNative.cc add

JNIEXPORT jint JNICALLJava_com_projecttango_experiments_nativehellotango_TangoJNINative_connectOnFrameAvailableCallback(
JNIEnv*, jobject) 
{
    return static_cast<int>(tango_handler.ConnectYUVFrameCallback());
}

In TangoJNINative.java add

// Connect the onFrameAvailable callback.
public static native int connectOnFrameAvailableCallback();

In HelloTangoActivity.java modify onResume()

protected void onResume() {
   super.onResume();
   // Setup Tango configuraturation.
   TangoJNINative.setupConfig();
   int status = 0;
   TangoJNINative.connect();
   status = TangoJNINative.connectOnFrameAvailableCallback();
   mIsTangoServiceConnected = true;
}

Endomorphic

Answer 2

Here is my code for converting an NV21 to an RGB frame. Maybe it is of any use...

static void
  cb_onFrameAvailable
  (
    void*                     contextA,
    TangoCameraId             idA,
    const TangoImageBuffer*  imageBufferA
  )
{
  // --- local constants ------------------------------

  // image width and height
  const int W               = imageBufferA->width;
  const int H               = imageBufferA->height;

  // sizes of Y, U, and V pixel arrays.
  const int sizeOfYDataL   = W * H;

  // indices, marking the begin of the y, u, and v data in the pixel buffer.
  const int beginOfYDataL  = 0;
  const int beginOfUVDataL  = sizeOfYDataL;

  // YUV, Y, and UV pixel sub arrays.
  const byte*  yuvArrL     = imageBufferA->data;
  const byte*  yArrL       = &yuvArrL[ beginOfYDataL  ];
  const byte*  uvArrL      = &yuvArrL[ beginOfUVDataL ];

  // --- local variables ------------------------------

  // image pixel coordinates.
  int xL,yL;

  // halved image pixel coordinates.
  int hxL,hyL;

  // ARGB value.
  int argbL;

  // --------------------------------------------------

  // translate YUV NV21 -> ARGB, using
  //
  //      / R \   / 1.000   0.000   1.596 \   /   Y   \
  //      | G | = | 1.000  -0.391  -0.813 | * | U-128 |
  //      \ B /   \ 1.000   2.018   0.000 /   \ V-128 /
  //

  // Note: start value yL=1 as the first scan line of the color image is ..
  //       .. reserved for metadata instead of image pixels.

  for( yL=1,hyL=0; yL<H; yL++,hyL=yL>>1 )
  {
    for( xL=0,hxL=0; xL<W; xL++,hxL=xL>>1 )
    {
      const int y = static_cast<int>( yArrL [  yL*W +    xL   ] )      ;
      const int v = static_cast<int>( uvArrL[ hyL*W + 2*hxL   ] ) - 128;
      const int u = static_cast<int>( uvArrL[ hyL*W + 2*hxL+1 ] ) - 128;

      int R = static_cast<int>( y               + ( 1.596f*v) );
      int G = static_cast<int>( y + (-0.391f*u) + (-0.813f*v) );
      int B = static_cast<int>( y + ( 2.018f*u)               );

      // clip RGB values to [0..255].
      R = R < 0 ? 0 : (R > 255 ? 255 : R);
      G = G < 0 ? 0 : (G > 255 ? 255 : G);
      B = B < 0 ? 0 : (B > 255 ? 255 : B);

      // combine to ARGB value.
      argbL = 0xff000000 | (R << 16) | (G << 8) | B;
    } // for
  } // for
} // function

Micronesia