How to get Audio for encoding using Xuggler

Question

I'm writing an application that records the screen and audio. While the screen recording works perfectly, I'm having difficulty in getting the raw audio using the JDK libraries. Here's the code:

try {
            // Now, we're going to loop
            long startTime = System.nanoTime();

            System.out.println("Encoding Image.....");
            while (!Thread.currentThread().isInterrupted()) {
                // take the screen shot
                BufferedImage screen = robot.createScreenCapture(screenBounds);


                // convert to the right image type
                BufferedImage bgrScreen = convertToType(screen,
                        BufferedImage.TYPE_3BYTE_BGR);

                // encode the image
                writer.encodeVideo(0, bgrScreen, System.nanoTime()
                        - startTime, TimeUnit.NANOSECONDS);

                /* Need to get audio here and then encode using xuggler. Something like 

                    WaveData wd = new WaveData();

                    TargetDataLine line;
                    AudioInputStream aus = new AudioInputStream(line);

                    short[] samples = getSourceSamples();
                       writer.encodeAudio(0, samples); */


                if (timeCreation < 10) {
                    timeCreation = getGMTTime();
                }
                // sleep for framerate milliseconds
                try {
                    Thread.sleep((long) (1000 / FRAME_RATE.getDouble()));
                } catch (Exception ex) {
                    System.err.println("stopping....");
                    break;
                }

            }
            // Finally we tell the writer to close and write the trailer if
            // needed
        } finally {
            writer.close();
        }

This page has some pseudo code like

while(haveMoreAudio())
 {
   short[] samples = getSourceSamples();
   writer.encodeAudio(0, samples);
 }

but what exactly should I do for getSourceSamples()?

Also, a bonus question - is it possible to choose from multiple microphones in this approach?

See also: Xuggler encoding and muxing

Hutton

Answer 1

Try this:

// Pick a format. Need 16 bits, the rest can be set to anything
// It is better to enumerate the formats that the system supports, because getLine() can error out with any particular format
AudioFormat audioFormat = new AudioFormat(44100.0F, 16, 2, true, false); 

// Get default TargetDataLine with that format
DataLine.Info dataLineInfo = new DataLine.Info( TargetDataLine.class, audioFormat );
TargetDataLine line = (TargetDataLine) AudioSystem.getLine(dataLineInfo);

// Open and start capturing audio    
line.open(audioFormat, line.getBufferSize());
line.start();

while (true) {
    // read as raw bytes
    byte[] audioBytes = new byte[ line.getBufferSize() / 2 ]; // best size?
    int numBytesRead = 0;
    numBytesRead =  line.read(audioBytes, 0, audioBytes.length);

    // convert to signed shorts representing samples
    int numSamplesRead = numBytesRead / 2;
    short[] audioSamples = new short[ numSamplesRead ];
    if (format.isBigEndian()) {
        for (int i = 0; i < numSamplesRead; i++) {
            audioSamples[i] = (short)((audioBytes[2*i] << 8) | audioBytes[2*i + 1]);
        }
    }
    else {
        for (int i = 0; i < numSamplesRead; i++) {
            audioSamples[i] = (short)((audioBytes[2*i + 1] << 8) | audioBytes[2*i]);
        }
    }

    // use audioSamples in Xuggler etc
}

To pick a microphone, you'd probably have to do this:

Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();
// Look through and select a mixer here, different mixers should be different inputs
int selectedMixerIndex = 0;
Mixer mixer = AudioSystem.getMixer(mixerInfo[ selectedMixerIndex ]);
TargetDataLine line = (TargetDataLine) mixer.getLine(dataLineInfo);

I think it's possible that multiple microphones will show up in one mixer as different source data lines. In that case you'd have to open them and call dataLine.getControl(FloatControl.Type.MASTER_GAIN).setValue( volume ); to turn them on and off.

See: WaveData.java

Sound wave from TargetDataLine

How to set volume of a SourceDataLine in Java

Bogeyman

Thanks. I'm guessing format should be audioFormat ? And what is data? Moreover, the audioSamples[i] = audioBytes[2 * i] << 8 | (0xFF & audioBytes[2 * i + 1]); needs a cast to short – Hutton

Thanks. But I get this error, after a couple of warnings. I've seen them before in previous attempts as well. an error occurred: error Operation not permitted, failed to write trailer to /path/to/outputfile.mp4 – Hutton

@wrahool: That error doesn't sound related to reading the audio. I take it that the audio code is now working? Please remember to accept and upvote, and post another question :) – Bogeyman

when I debug the code, I check the values in the array audioSamples and they are all -2. I'm using this array to encode the audio, could it be that the error is owing to that? – Hutton

@wrahool: I doubt it. Try setting audioSamples to all zeros after reading but just before encode, it should have no effect on the error. I have an idea about that error, but - please post the code you use to write the file in another question. – Bogeyman

@wrahool: If audioSamples is -2, what is audioBytes? – Bogeyman

@wrahool: Anyway, this is a pretty complete example. If you want me to debug it as well, post a bounty :) – Bogeyman