What is the fastest way to create a 2GB file containing random bytes in Java?

Question

I want a way to generate a file containing random bits in Java. What will create the random file the fastest? I want to create files of any specified size containing random bits. I want to be able to generate a 2GB file in a matter of minutes (less than a minute if possible). The technique I'm using right now takes hours to do 2GB:

...
private static Random r = new Random();

private static int rand(int lo, int hi) {
    int n = hi - lo + 1;
    int i = r.nextInt() % n;
    if (i < 0) {
        i = -i;
    }
    return lo + i;
}
...
fos = new FileOutputStream(hdFiller);
for(long i = 0; i < maxFileSize; i++) {
    int idx = rand(0,32);
    fos.write(idx);
}
fos.close();
...

There has got to be a way to do this faster, maybe even in less than a minute for 2GB.

Thanks.

Reprobate

What platform are you on? If it's unix-ish, you'd be better off doing this at the command line: dd if=/dev/urandom of=random.txt bs=4096 count=512 (4096 * 512 = 2meg, adjust as needed for your 2gig). – Bazemore

How random is random? The more random, the longer it takes. Why does it need to be random at all? – Tessy

Answer 1

If you want to generate random bits all at once, rather than looping, take a look at the java.util.Random method nextBytes(byte[]) which fills the specified byte array with random bytes. Create a byte array exactly large enough for 2GiB of data and you can generate the whole random bit source in one go.

Putt

Don't do this literally as your program will be needing 2GB of memory, and most likely crash due to out of heap memory error. Instead produce the byte array in smaller chunks. – Tequilater

Somewhat smaller chunks would probably be better, and write it using a java.nio FileChannel, possibly as a MemoryMappedFile (Thinking in Java Memory-mapped files tutorial) – Ferrer

For a case where each file location is accessed only once the memory-mapped file may not offer much advantage and could cause excessive MMU work. All that is needed for top performance is a direct byte buffer. – Doolittle

Answer 2

Try wrapping your FileOutputStream with a BufferedOutputStream.

Webworm

Answer 3

I've created this method to generate Files with random content.

Memory consumption is minimal, as we do not keep the whole file in memory, but write it to the disk through an 8K buffer.

The file will be in the system's temp folder, and will be deleted automatically when the JVM stops. (I use it in tests only, but can be modified if auto cleanup is not necessary.)

An example usage: generateFile(ofGigabytes(2)). (If you don't use spring, just use long instead of DataSize class as the function argument).

import lombok.SneakyThrows;
import org.jetbrains.annotations.NotNull;
import org.springframework.util.unit.DataSize;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.util.Random;

import static org.springframework.util.unit.DataSize.ofKilobytes;

public class DocumentTestUtil {

    private static final long DEFAULT_BUFFER_SIZE = ofKilobytes(8).toBytes();

    private DocumentTestUtil() {
    }

    @SneakyThrows
    public static @NotNull File generateFile(DataSize size) {
        File file = File.createTempFile("test-document-", null);
        file.deleteOnExit();
        try (final var os = new BufferedOutputStream(new FileOutputStream(file))) {
            long numOfBytes = 0;
            final var bytes = new byte[calculateBufferSize(size)];
            final var random = new Random();
            while (numOfBytes < size.toBytes()) {
                random.nextBytes(bytes);
                os.write(bytes);
                numOfBytes += bytes.length;
            }
        }
        return file;
    }

    private static int calculateBufferSize(final DataSize size) {
        if (size.toBytes() > DEFAULT_BUFFER_SIZE) {
            return (int) DEFAULT_BUFFER_SIZE;
        }
        return (int) size.toBytes();
    }

}

Klinges