Reliable sound API in Java for simple digital samples playback

Question

Is there a good recipe to get decent, reliable digital sampled sound playback in Java?

My list of requests is pretty short:

• Load digitized samples in memory (for example, from resouces bundled in jar) from something like .wav files
• Play them in non-blocking manner
• When I play several samples simultaneously and they intersect in time, they should get properly mixed

It would be nice to have the following, but in fact I can live without it:

• Playing from .ogg or similar compressed format (without implementing a CPU-hungry decoder in Java, obviously)
• Playing back the same sample again while it is still playing should not stop previous playback of a given sample, but second copy should start and get properly mixed with the first one

I've tried the infamous Java Sound API, but found out that it is utterly unreliable and seem to be unable to satisfy even my minimal wish list. The problems I get:

• On Linux with ALSA dmix (OpenJDK 6), having any other application using audio while initializing Java Sound API just makes all the sound from Java app disappear without any errors / warnings.

• On Linux (OpenJDK 6), listing MixerInfos and trying to get a Clip object using any of them throws the following exception when trying to load a wav file:

  java.lang.IllegalArgumentException: Line unsupported: interface Clip supporting format PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian
  

  Thus, AudioSystem.getClip(anySortOfMixer) doesn't seem to work at all. Only AudioSystem.getClip() works.

• Loading files with different sample rate / bits / format using Clip fails with LineUnavailableException. It seems that first invocation of clip.open sets up sound system to a particular sound options, following invocations to load a file with slightly different sample rate (for example, first one was 44100, second one is 48000)

• On Linux (OpenJDK 6) initializing several different Clips and trying to play them makes only last loaded Clip audible - no errors/warnings are given, but only using play on last Clip loaded makes any sound at all - all others are silent:

  Clip loadSound(String name) {
      URL url = this.getClass().getResource("/" + name + ".wav");
      Clip clip = AudioSystem.getClip();
      AudioInputStream ais = AudioSystem.getAudioInputStream(url);
      clip.open(ais);
      return clip;
  }
  
  void playSound(Clip) {
      if (clip.isRunning())
          clip.stop();
      clip.setFramePosition(0);
      clip.start();
  }
  ...
  Clip c1 = loadSound("foo");
  Clip c2 = loadSound("bar");
  ...
  playSound(c1); // silence
  ...
  playSound(c2); // audible
  

  Everything's fine with this code on Windows - all Clips are audible, play and mix properly. Haven't tested it on Mac.

• Supported file formats (analyzed with AudioSystem.getAudioFileTypes) returns wav / au / aif on both Linux/OpenJDK6 and Windows/Oracle JDK 7, so no oggs or even mp3s :(

• There seem to be no easy way to make two copies of the same Clip sound simultaneously without loading 2nd copy as a distinct Clip.

So, the question is - is there a good solution / workaround to remedy all this stuff and make it more reliable? Would switching to some other sound system (such as LWJGL OpenAL or paulscode.com sound system) help? Or is it possible to wrap Java Sound API in some safe-guards and it will work properly?

I've made a little application that tests all of the above, but it's a bit long, so I thought I'd publish it as a gist, but, unfortunately, GitHub is having some network issues right now. So, I guess, I will publish it a bit later.

Sadomasochism

"simple digital samples" ".ogg or similar compressed format" Compressed sound formats are not simple. -- Perhaps JavaFX offers a suitable control or API. Good question BTW. +1 – Felishafelita

It depends on a definition of simple ;) Generally, if I understand correctly, even in Java Sound API, one can plug in a different decoder for formats such as mp3 or ogg, but I haven't even explored this opportunity as it is too much hassle for my case. I'd definitely be happy with wavs only. – Sadomasochism

"I'd definitely be happy with wavs only." (shrugs) If you can find an API that does sound reliably, I suspect it will support compressed formats. I've previously added the mp3plugin.jar of the JMF for MP3 support in Java Sound apps. I suggested JavaFX in the hope it uses some entirely alternate API to Java Sound (which, as you note, as less than entirely reliable). But speaking of Linux & sound, lack of sound support on Linux was the thing that caused Jamie Zawinski to toss Linux (for which he had written a lot of code) for Mac. ;) – Felishafelita

I personally like using OpenAL via LWJGL – Hilary

You could try gstreamer for Java. I've used it in the past, it takes some effort understanding how to set up the audio pipeline, but once you get it running it should be able to handle almost anything you throw at it. code.google.com/p/gstreamer-java – Saxtuba

Answer 1

I posted a fairly simple, limited, audio mixer at Java-gaming.org, which you are welcome to check out at the following url: http://www.java-gaming.org/topics/simple-audio-mixer-2nd-pass/27943/view.html

The jar listed in the first post has source code and sample usages and I put some energy into making javadoc comments. (98% of the download is the single sample wav I included.) Also, on the thread is a lot of api info.

It has issues with Linux, still. But I am impressed with your analysis, and am wondering about sharing in the effort to try and troubleshoot and fix this!

On your points:

• I recall hearing that with some Linux systems, a single output is all that is possible, and that some applications do not play fair and release the audio to Java when there is contention. To the extent that this is accurate, it would be hard to call this a Java problem, but rather perhaps a Linux OS problem?

• Second point: I haven't tried loading from a Mixer in Linux yet, but I know some folks have been able to do this from my web app Java Theremin. In that app (linked in the above thread) I include a dropdown that allows the user to choose the mixer. At least some of the Linux users have had success with this.

• I haven't used Big-Endian wavs -- but only little endian wavs. You'd have to flip the bytes in Audacity or something similar to use my mixer as it currently stands.

• My system DOES handle concurrency. You load a wav into a PFClipData object. Then, you can play this back a number of different ways, via a PFClipShooter (can handle concurrent playbacks--20 or 30, and at different pitches as well) or a PFClipLooper (will loop the clip with optional overlap modes of the end to help smooth out the looping point). ALL output is funneled into a single SourceDataLine behind the scenes.

• I haven't implemented ogg or mp3 yet, just 16-bit, 44100fps stereo little-endian wav files.

• Would be happy to consider making this an open source git project, if there were others willing to share in this.

--I succeeded in installing Linux (Ubuntu Desktop) in a dual boot partition on my PC very recently, and am about to install a sound card and see if I recreate and hopefully fix some of the problems that are being described. The Ubuntu has both OpenJDK and Oracle's JDK, so I hope to see if the Java implementation might be part of the issue. Work in progress...

Atrocious

Thanks, Phil! I'll definitely see into your solution and will report a bit later. – Sadomasochism

I've took a look and, basically, I see that you've implemented a software mixer yourself. You sum up data coming from PFMixerTracks in a loop in PFAudioMixer - that's good, but it misses out a few important points in my opinion: 1) good signal mixing is much more intricate than just summing everything up (and getting weird clipping artifacts), 2) signal mixing is mostly done in hardware on modern sound cards, 3) even if it's not hardware mixing, doing such CPU-intensive job in pure Java is pretty sub-optimal. – Sadomasochism

I see your point: you're doing do-all-the-stuff-without-extra-dependencies-and-complexity-so-it-will-work-everywhere solution. However, I'm not sure that falling back to software mixing in pure Java worth it. I'll remember your solution and will use it if I couldn't find a pack of workarounds to increase JavaSound's own reliability. – Sadomasochism

Please post a reply here if you come across an answer!! I think in your second comment you get the drift of my implementation: I don't worry about clipping because that can/should be handled by the programmer doing basic quality control on levels BEFORE releasing the app. I agree hardware mixing would be preferable. Last point, I think the cost to the cpu is less than is normally imagined, that a LOT can be done with as little as 2% of its capacity. Thanks for taking a look at and commenting on the code. – Atrocious

Technically, proper audio signals mixing is a much harder task than just adding individual samples as is. There are a dozen of methods, most of them are pretty CPU-hungry (such as doing hi-res FFT, adding up spectrums, then doing inverse FFT on result). Most of the methods can achieve decent sound levels without clipping and even without resorting to extra clipping removal methods, such as limiting. For example, you can improve your summing by using res = A + B - A * B technique, as described in vttoth.com/CMS/index.php/technical-notes/68 – Sadomasochism

Thanks for link. Neat article! But for my simple, limited game scenarios, I will prefer to figure out beforehand what volumes not to exceed, and avoid the cross-product calculations. Can you imagine using his method for the 32 clips that are all played at the same time in my demo? I don't see a simple game needing FFT-level processing. Maybe a low pass filter, for rolling off highs for distance effects, but even that doesn't need more than a few poles (i.e., three or five samples of history, perhaps). Trying to keep things to bare minimum! Better to do complicated stuff in a proper DAW. – Atrocious