Use rotateM() of Matrix to rotate matrix from SurfaceTexture but corrupt the video output

Question

I managed to play video with opengl es, I used the way of grafika's ContinuousCaptureActivity, my data source is MediaPlayer rather than Camera which makes no difference. MediaPlayer produces video frames continuously and I draw each frame to screen in onFrameAvailable callback. The code is as follows which works well:

    mVideoTexture.updateTexImage();
    mVideoTexture.getTransformMatrix(mTmpMatrix);
    mDisplaySurface.makeCurrent();
    int viewWidth = getWidth();
    int viewHeight = getHeight();
    GLES20.glViewport(0, 0, viewWidth, viewHeight);
    mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
    mDisplaySurface.swapBuffers();

Now I want to rotate video frames with 270 degrees, so I changed the code:

        GLES20.glClearColor(1.0f, 0.0f, 0.0f, 0.0f);
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

    mVideoTexture.updateTexImage();
    mVideoTexture.getTransformMatrix(mTmpMatrix);
    mDisplaySurface.makeCurrent();
    int viewWidth = getWidth();
    int viewHeight = getHeight();
    GLES20.glViewport(0, 0, viewWidth, viewHeight);
    Matrix.rotateM(mTmpMatrix, 0, 270, 1f, 0, 0);
    mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
    mDisplaySurface.swapBuffers();

But the result is weird, take a look at the picture below:

But I can flip video frame successfully with the code below:

        mVideoTexture.updateTexImage();
    mVideoTexture.getTransformMatrix(mTmpMatrix);
    mDisplaySurface.makeCurrent();
    int viewWidth = getWidth();
    int viewHeight = getHeight();
    GLES20.glViewport(0, 0, viewWidth, viewHeight);
    mTmpMatrix[5] = -1 * mTmpMatrix[5];
    mTmpMatrix[13] = 1.0f - mTmpMatrix[13];
    mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
    mDisplaySurface.swapBuffers();

How to achieve the rotation, Could anyone give me some help?

ADD:

At first, I want to tell that I always used this code for each draw action:

        GLES20.glClearColor(1.0f, 0.0f, 0.0f, 0.0f);
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

I use this demo to do my test, it is a very good demo for this test. https://github.com/izacus/AndroidOpenGLVideoDemo

The matrix got from the surfacetexture is:

1.0, 0.0, 0.0, 0.0

0.0, -1.0, 0.0, 0.0

0.0, 0.0, 1.0, 0.0

0.0, 1.0, 0.0, 1.0

After "Matrix.setRotateM(videoTextureTransform, 0, 270 , 0, 0, 1);",

it became:

1.1924881E-8, -1.0, 0.0, 0.0

1.0, 1.1924881E-8, 0.0, 0.0

0.0, 0.0, 1.0, 0.0

0.0, 0.0, 0.0, 1.0

And this video effect is:

Limitative

@Reto Koradi Hi, Could you help me, please? – Limitative

Answer 1

fadden's rotation matrix about the z-axis needs to be followed by the correct translation to bring it back on-screen, so to speak. I've tested all 3 rotations below on SurfaceTexture video:

ROTATE 90

Matrix.rotateM(mTmpMatrix, 0, 90, 0, 0, 1);
Matrix.translateM(mTmpMatrix, 0, 0, -1, 0);

ROTATE 180

Matrix.rotateM(mTmpMatrix, 0, 180, 0, 0, 1);
Matrix.translateM(mTmpMatrix, 0, -1, -1, 0);

ROTATE 270

Matrix.rotateM(mTmpMatrix, 0, 270, 0, 0, 1);
Matrix.translateM(mTmpMatrix, 0, -1, 0, 0);

Preen

It works! Thanks a lot. But why did you use -1 for every translation? Let's talk about rotation 90 degree about z axis, rotating 90 will move the texture coordinate to the bottom of y axis, I think we should use 1 to translate it. Could you explain the reason for me? – Limitative

How would I flip the image over the X axis? I'm using the rotate 180 version but the image is mirrored, so I would prefer to avoid the rotation. – Psephology

I heartily thank you bro. this word is to short for you but Thank you so much @Preen – Sentimental

I understood， this is texture coordinate, if just rotate, the image will be out of screen. So do translation at first then rotate. – Limitative

Answer 2

You're rotating about the X axis:

Matrix.rotateM(mTmpMatrix, 0, 270, 1f, 0, 0);

By convention that runs from left to right. The axis acts like an axle; by flipping it 270 degrees, you're rotating the plane so you're viewing it edge-on, and it's effectively vanishing. I think what you're seeing is essentially uninitialized data, and if you call glClear() you'll see the background color instead.

Try rotating about the Z axis, which is a line pointing out of the screen:

Matrix.rotateM(mTmpMatrix, 0, 270, 0, 0, 1);

(It might also be interesting to experiment with a rotation of about 15 degrees about the X axis just to see how that looks. When fiddling with matrices it's often useful to start with small values.)

Dmso

"Matrix.rotateM(mTmpMatrix, 0, 270, 0, 0, 1f)" does not help. The result is similar, the image is also weird just like mine. – Limitative

This demo used the same way to render video as mine. Testing with this demo will save lots of time and get the same effect. github.com/izacus/AndroidOpenGLVideoDemo – Limitative

What effect does glClear() have? Does applying a small rotation (say, 2 degrees) have the same effect? What are the contents of your matrix? – Dmso

Hi, I have updated my question and give the information that you want. Please have a look. I found that the video rotation is not an easy task! – Limitative

a small degree has a different effect. We can see the content of the video but the direction is weird. – Limitative

Forget that it's video -- the sample source shouldn't matter. This is just about transforming the vertices of a GLES polygon. Does it look reasonable if you start with the "identity" matrix rather than the matrix from the SurfaceTexture? I don't see an issue with the matrix being used, but I'm far from expert on these matters. (If it does work with identity, you could work around the issue by flipping the image by inverting the texture V coordinates rather than flipping the vertices.) I don't think it's GLES20.glDisable(GLES20.GL_CULL_FACE) -- should see nothing if that's an issue. – Dmso

If I use rotateM() to modify the "identity" matrix (mvpMatrix) rather than the matrix from the SurfaceTexture. The video frame can rotate succefully with 90,180,270. This is a good way, but I want to modify the matrix from the SurfaceTexture to achieve rotation because my video has angle. The rotation will change the position of the angle if I modify the identity matrix. – Limitative

I believe there is way to achieve the rotation of matrix from surfaceTexture. – Limitative

Right now you're calculating flip * rotation. Have you tried rotation * flip instead? – Dmso

Do you mean the identity matrix or the matrix from SurfaceTexture? – Limitative

I mean the matrix from the SurfaceTexture. The order in which matrices are multiplied is significant. It's been a while, but I believe you want to multiply them in the opposite order in which the operations should be applied. Since you want to flip then rotate, you would start with the rotation matrix and then multiply by the flip matrix. Multiplying by the identity matrix has no effect either way. (I should point out: rotateM() creates a temporary matrix, populates it with setRotateM(), then calls multiplyMM() to apply it.) – Dmso

Thanks very much,I will try it later. – Limitative

Maybe rotating matrix never works for me. Scaling may help according to this topic: blender.stackexchange.com/questions/5608/rotate-object-texture – Limitative