Video in Android : change visual properties (e.g. saturation, brightness)

Question

Assuming we have a Surface in Android that displays a video (e.g. h264) with a MediaPlayer:

1) Is it possible to change the displayed saturation, contrast & brightness of the displayed on the surface video? In real time? E.g. Images can use setColorFilter is there anything similar in Android to process the video frames?

Alternative question (if no. 1 is too difficult):

2) If we would like to export this video with e.g. an increased saturation, we should use a Codec, e.g. MediaCodec. What technology (method, class, library, etc...) should we use before the codec/save action to apply the saturation change?

Gasaway

Do you want a ready-made library that can manipulate video for display (as in the answer below), or do you have other requirements that don't permit that? The linked project in the answer demonstrates using a GLSurfaceView with a MediaPlayer and applying OpenGL shaders to perform filtering on the display. That is definitely a reasonable approach if you don't need to export the filtered video. Where is your Surface coming from now? – Aleshia

Hi @Dave, I am currently investigating how it's possible to both adjust the displayed video (usually on the surface) but also export the adjusted video. So I have no concrete Surface - that's what I am looking for :) – Gasaway

Answer 1

For display only, one easy approach is to use a GLSurfaceView, a SurfaceTexture to render the video frames, and a MediaPlayer. Prokash's answer links to an open source library that shows how to accomplish that. There are a number of other examples around if you search those terms together. Taking that route, you draw video frames to an OpenGL texture and create OpenGL shaders to manipulate how the texture is rendered. (I would suggest asking Prokash for further details and accepting his answer if this is enough to fill your requirements.)

Similarly, you could use the OpenGL tools with MediaCodec and MediaExtractor to decode video frames. The MediaCodec would be configured to output to a SurfaceTexture, so you would not need to do much more than code some boilerplate to get the output buffers rendered. The filtering process would be the same as with a MediaPlayer. There are a number of examples using MediaCodec as a decoder available, e.g. here and here. It should be fairly straightforward to substitute the TextureView or SurfaceView used in those examples with the GLSurfaceView of Prokash's example.

The advantage of this approach is that you have access to all the separate tracks in the media file. Because of that, you should be able to filter the video track with OpenGL and straight copy other tracks for export. You would use a MediaCodec in encode mode with the Surface from the GLSurfaceView as input and a MediaMuxer to put it all back together. You can see several relevant examples at BigFlake.

You can use a MediaCodec without a Surface to access decoded byte data directly and manipulate it that way. This example illustrates that approach. You can manipulate the data and send it to an encoder for export or render it as you see fit. There is some extra complexity in dealing with the raw byte data. Note that I like this example because it illustrates dealing with the audio and video tracks separately.

You can also use FFMpeg, either in native code or via one of the Java wrappers out there. This option is more geared towards export than immediate playback. See here or here for some libraries that attempt to make FFMpeg available to Java. They are basically wrappers around the command line interface. You would need to do some extra work to manage playback via FFMpeg, but it is definitely doable.

If you have questions, feel free to ask, and I will try to expound upon whatever option makes the most sense for your use case.

Aleshia

Hi Dave, you answer definitely deserves the bounty as it's very analytical. The most balanced solution to create both visual filters and exporting them looks manual work with MediaCodec. For sure I am looking if I can use FFMpeg (which looks awesome) with some "preview" possibility too. Thanks! – Gasaway

Answer 2

If you are using a player that support video filters then you can do that.

Example of such a player is VLC, which is built around FFMPEG [1].

VLC is pretty easy to compile for Android. Then all you need is the libvlc (aar file) and you can build your own app. See compile instructions here.

You will also need to write your own module. Just duplicate an existing one and modify it. Needless to say that VLC offers strong transcoding and streaming capabilities.

As powerful VLC for Android is, it has one huge drawback - video filters cannot work with hardware decoding (Android only). This means that the entire video processing is on the CPU.

---

Your other options are to use GLSL / OpenGL over surfaces like GLSurfaceView and TextureView. This guaranty GPU power.

Flavio