How can I turn a series of images into a video using JCodec?

Question

I'm trying to use JCodec to turn a series of images into a video inside of a Java SE desktop application. The few methods I've tried all resulted in a video that Windows Media Player could not play.

It is unclear to me if this is a codec issue (doubtful) or if I'm not creating the video properly. When I try to play the video in Windows Media Player I get:

Windows Media Player cannot play the file. The Player might not support the file type or might not support the codec that was used to compress the file.

Here is the cobbled together sample that I've been using most recently. I really don't understand the internals of the video format, so I'm not even entirely sure what some of the code is doing.

import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;

import javax.imageio.ImageIO;

import org.jcodec.codecs.h264.H264Encoder;
import org.jcodec.codecs.h264.H264Utils;
import org.jcodec.common.NIOUtils;
import org.jcodec.common.SeekableByteChannel;
import org.jcodec.common.model.ColorSpace;
import org.jcodec.common.model.Picture;
import org.jcodec.containers.mp4.Brand;
import org.jcodec.containers.mp4.MP4Packet;
import org.jcodec.containers.mp4.TrackType;
import org.jcodec.containers.mp4.muxer.FramesMP4MuxerTrack;
import org.jcodec.containers.mp4.muxer.MP4Muxer;
import org.jcodec.scale.AWTUtil;
import org.jcodec.scale.RgbToYuv420;

public class CreateVideo {
    private SeekableByteChannel ch;
    private Picture toEncode;
    private RgbToYuv420 transform;
    private H264Encoder encoder;
    private ArrayList<ByteBuffer> spsList;
    private ArrayList<ByteBuffer> ppsList;
    private FramesMP4MuxerTrack outTrack;
    private ByteBuffer _out;
    private int frameNo;
    private MP4Muxer muxer;

    public CreateVideo(File out) throws IOException {
        ch = NIOUtils.writableFileChannel(out);

        // Transform to convert between RGB and YUV
        transform = new RgbToYuv420(0, 0);

        // Muxer that will store the encoded frames
        muxer = new MP4Muxer(ch, Brand.MP4);

        // Add video track to muxer
        outTrack = muxer.addTrackForCompressed(TrackType.VIDEO, 25);

        // Allocate a buffer big enough to hold output frames
        _out = ByteBuffer.allocate(1920 * 1080 * 6);

        // Create an instance of encoder
        encoder = new H264Encoder();

        // Encoder extra data ( SPS, PPS ) to be stored in a special place of
        // MP4
        spsList = new ArrayList<ByteBuffer>();
        ppsList = new ArrayList<ByteBuffer>();

    }

    public void encodeImage(BufferedImage bi) throws IOException {
        if (toEncode == null) {
            toEncode = Picture.create(bi.getWidth(), bi.getHeight(), ColorSpace.YUV420);
        }

        // Perform conversion
        for (int i = 0; i < 3; i++) {
            Arrays.fill(toEncode.getData()[i], 0);
        }
        transform.transform(AWTUtil.fromBufferedImage(bi), toEncode);

        // Encode image into H.264 frame, the result is stored in '_out' buffer
        _out.clear();
        ByteBuffer result = encoder.encodeFrame(_out, toEncode);

        // Based on the frame above form correct MP4 packet
        spsList.clear();
        ppsList.clear();
        H264Utils.encodeMOVPacket(result, spsList, ppsList);

        // Add packet to video track
        outTrack.addFrame(new MP4Packet(result, frameNo, 25, 1, frameNo, true, null, frameNo, 0));

        frameNo++;
    }

    public void finish() throws IOException {
        // Push saved SPS/PPS to a special storage in MP4
        outTrack.addSampleEntry(H264Utils.createMOVSampleEntry(spsList, ppsList));

        // Write MP4 header and finalize recording
        muxer.writeHeader();
        NIOUtils.closeQuietly(ch);
    }

    public static void main(String[] args) throws IOException {
        CreateVideo encoder = new CreateVideo(new File("C:\\video\\video.mp4"));
        for (int i = 10; i < 34; i++) {
            String filename = String.format("C:\\video\\%02d.png", i);
            BufferedImage bi = ImageIO.read(new File(filename));
            encoder.encodeImage(bi);
        }
        encoder.finish();
    }
}

I'm not tied to H264 or MP4 if there is some easier codec/container. The only requirement is that it should play on a baseline Windows 7 box with no additional software installed.

Mchail

Answer 1

I had a lot of issues with jcodec's RgbToYuv420() transform.
I used my own function to transform my RGB BufferedImage to Yuv420.
One issue I had was that RgbToYuv420 averages the computed upix and vpix values, and it was causing the colors in my mp4 to be all wonky.

// make a YUV420J out of BufferedImage pixels
private Picture makeFrame(BufferedImage bi, ColorSpace cs)
{   
    DataBuffer imgdata = bi.getRaster().getDataBuffer();
    int[] ypix = new int[imgdata.getSize()];
    int[] upix = new int[ imgdata.getSize() >> 2 ];
    int[] vpix = new int[ imgdata.getSize() >> 2 ];
    int ipx = 0, uvoff = 0;

    for (int h = 0; h < bi.getHeight(); h++) {
        for (int w = 0; w < bi.getWidth();  w++) {
            int elem = imgdata.getElem(ipx);
            int r = 0x0ff & (elem >>> 16);
            int g = 0x0ff & (elem >>> 8);
            int b = 0x0ff & elem;
            ypix[ipx] = ((66 * r + 129 * g + 25 * b) >> 8) + 16;
            if ((0 != w % 2) && (0 != h % 2)) {
                upix[uvoff] = (( -38 * r + -74 * g + 112 * b) >> 8) + 128;
                vpix[uvoff] = (( 112 * r + -94 * g + -18 * b) >> 8) + 128;
                uvoff++;
            }
            ipx++;
        }
    }
    int[][] pix = { ypix, upix, vpix, null };
    // old API
    // return new Picture(bi.getWidth(), bi.getHeight(), pix, ColorSpace.YUV420J);
    return Picture.createPicture(bi.getWidth(), bi.getHeight(), pix, ColorSpace.YUV420J);
}

Shortly

That Picture constructor no longer exists. – Gratify

edited to remove new Picture() and replace with Picture.creatPicture() – Shortly

@Shortly I just tested JCodec from source and I don't see any color shift anymore. Seems a big commit was made in 2016 (2 years after your initial answer) regarding "bugs and color problems" - github.com/jcodec/jcodec/commit/… - Care to check if your fix is still relevant with the current version ? Thanks. – Upstate

Answer 2

With 0.2.0 there is now https://github.com/jcodec/jcodec/blob/master/javase/src/main/java/org/jcodec/api/awt/AWTSequenceEncoder8Bit.java

This example creates a MP4 with a fixed background and overlaid animation. No sound.

    AWTSequenceEncoder8Bit enc = AWTSequenceEncoder8Bit.create2997Fps(outputMovieFile); 
    int framesToEncode = (int) (29.97 * durationInSeconds);
    java.awt.image.BufferedImage image = ...(some background image)

    for (int frameIndx = 0, x = 0, y = 0, incX = speed, incY = speed; frameIndx < framesToEncode; frameIndx++, x += incX, y += incY) {
        Graphics g = image.getGraphics();
        g.setColor(Color.YELLOW);
        if (x >= image.getWidth() - ballSize) incX = -speed;
        if (y >= image.getHeight() - ballSize) incY = -speed;
        if (x <= 0) incX = speed;
        if (y <= 0) incY = speed;
        g.fillOval(x, y, ballSize, ballSize);
        enc.encodeImage(image);
    }
    enc.finish();

Gable