Appending files to a zip file with Java

Question

I am currently extracting the contents of a war file and then adding some new files to the directory structure and then creating a new war file.

This is all done programatically from Java - but I am wondering if it wouldn't be more efficient to copy the war file and then just append the files - then I wouldn't have to wait so long as the war expands and then has to be compressed again.

I can't seem to find a way to do this in the documentation though or any online examples.

Anyone can give some tips or pointers?

UPDATE:

TrueZip as mentioned in one of the answers seems to be a very good java library to append to a zip file (despite other answers that say it is not possible to do this).

Anyone have experience or feedback on TrueZip or can recommend other similar libaries?

Aeneid

found this post in the truezip mailing list: truezip.dev.java.net/servlets/… conclusion: truezip currently does not support fast append operations – Parsons

Answer 1

In Java 7 we got Zip File System that allows adding and changing files in zip (jar, war) without manual repackaging.

We can directly write to files inside zip files as in the following example.

Map<String, String> env = new HashMap<>(); 
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env))
{
    Path nf = fs.getPath("new.txt");
    try (Writer writer = Files.newBufferedWriter(nf, StandardCharsets.UTF_8, StandardOpenOption.CREATE)) {
        writer.write("hello");
    }
}

Macronucleus

How can we use this one using smb? I want to add files to a zip file which is in a windows machine from a osx/linux machine. – Urina

@NirmalRaghavan This is out of scope of this question. For SMB/CIFS, see how to mount Windows network drive in Linux. – Sisal

Thanks for the example. Turns out I was to stupid to use ZIP-FileSystems till now. – Ardenia

Wow, that was it. Much easier than the current top answer, which seems somewhat outdated as best answer since Java 7. @Grouchal: can / would you revoke or move your +150 in order to boost this answer? (We just spent some hours trying to get TrueVFS to work in vain...) – Fate

Zip File System can't really deal with whitespaces in the folder structure. For a workaround, encode all whitespaces with "%2520" (see also #9874345 ) – Arwood

Beware that ZipFileSystem by default is vulnerable to OutOfMemoryError on huge inputs. – Gilligan

I am working with java 6 and i have similar kind of requirement.Moreover i have directories in zip file and i have to merge two zip files.Any help? – Palacio

Thank you very much. That led me into the right direction. I was using Java 8 and binary content and with the Files.newByteChannel method I had to provide the OpenOptions CREATE and WRITE otherwise an exception was thrown. – Stedt

Use Files.copy instead: try (FileSystem jarFs = FileSystems.newFileSystem(uri, env, null)) { for(final Path newFilePath : newFilePathList) { final Path pathInZipFile = jarFs.getPath("/" + newFilePath.getFileName()); Files.copy(newFilePath, pathInZipFile, StandardCopyOption.REPLACE_EXISTING); } } – Prostyle

This answer shows how to do it but how does it work under the covers? Is updating a file in a zip efficient or is it equivalent to unzipping and building a new zip? – Ecosystem

The [Zip File System][1] mentioned in this answer may be easy to use but, as others have queried, it's not entirely clear how it actually copies an entry from one zip to another (byte-for-byte or decompress/recompress?). [1]: docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/… – Dulci

Answer 2

As others mentioned, it's not possible to append content to an existing zip (or war). However, it's possible to create a new zip on the fly without temporarily writing extracted content to disk. It's hard to guess how much faster this will be, but it's the fastest you can get (at least as far as I know) with standard Java. As mentioned by Carlos Tasada, SevenZipJBindings might squeeze out you some extra seconds, but porting this approach to SevenZipJBindings will still be faster than using temporary files with the same library.

Here's some code that writes the contents of an existing zip (war.zip) and appends an extra file (answer.txt) to a new zip (append.zip). All it takes is Java 5 or later, no extra libraries needed.

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;

public class Main {

    // 4MB buffer
    private static final byte[] BUFFER = new byte[4096 * 1024];

    /**
     * copy input to output stream - available in several StreamUtils or Streams classes 
     */    
    public static void copy(InputStream input, OutputStream output) throws IOException {
        int bytesRead;
        while ((bytesRead = input.read(BUFFER))!= -1) {
            output.write(BUFFER, 0, bytesRead);
        }
    }

    public static void main(String[] args) throws Exception {
        // read war.zip and write to append.zip
        ZipFile war = new ZipFile("war.zip");
        ZipOutputStream append = new ZipOutputStream(new FileOutputStream("append.zip"));

        // first, copy contents from existing war
        Enumeration<? extends ZipEntry> entries = war.entries();
        while (entries.hasMoreElements()) {
            ZipEntry e = entries.nextElement();
            System.out.println("copy: " + e.getName());
            append.putNextEntry(e);
            if (!e.isDirectory()) {
                copy(war.getInputStream(e), append);
            }
            append.closeEntry();
        }

        // now append some extra content
        ZipEntry e = new ZipEntry("answer.txt");
        System.out.println("append: " + e.getName());
        append.putNextEntry(e);
        append.write("42\n".getBytes());
        append.closeEntry();

        // close
        war.close();
        append.close();
    }
}

Tigress

My war file is 30Mb compressed - not sure this approach will be the best way as it will require a lot of memory - I am already caching a lot of database queries in memory and this might make the memory footprint too big. – Aeneid

@Aeneid Actually you won't ever need more memory than BUFFER (I've chosen 4MB, but you're free to tailor it to your needs - it shouldn't hurt to reduce it to a few KB only). The file is never stored entirely in memory. – Tigress

the idea is to decompress contents of the existing war into BUFFER and compress it into a new archive - entry after entry. After that, you end up with the same archive that's ready to take some more entries. I've chosen to write "42" into answer.txt. That's where you should place your code to append more entries. – Tigress

How would this approach compare to using TrueZip - mentioned by gnlogic? TrueZip seems to really append to the file – Aeneid

Sorry, I didn't know this library. After digging through the code, I still don't know what it's doing, but it's doing it pretty fast :) So yes, it seem to really append content to a file. – Tigress

However, as you said you want "to copy the war file and then just append the files" I assume you don't want to modify the source. Using TrueZip, you'll have to copy the file which isn't necessary with the code above. Therefore, both approaches should finally be quite similar in performance. – Tigress

Truezip uses the concept of treating the zip file like a virtual file system. If you wanna copy and append - I bet that should be pretty easy too. – Exhale

If you get a ZipException - invalid entry compressed size with this approach, see coderanch.com/t/275390/Streams/java/… – Harbot

This code will only work for a zip file that was created by the java encoder. You can't reuse the ZipEntry because it stores the compressed size, which will depend on the compression settings. – Mason

Answer 3

I had a similar requirement sometime back - but it was for reading and writing zip archives (.war format should be similar). I tried doing it with the existing Java Zip streams but found the writing part cumbersome - especially when directories where involved.

I'll recommend you to try out the TrueZIP (open source - apache style licensed) library that exposes any archive as a virtual file system into which you can read and write like a normal filesystem. It worked like a charm for me and greatly simplified my development.

Exhale

This looks very good - would like to know if there are any performance issues to know about? – Aeneid

So far I've been able to use it effectively with moderately sized files (3 MB etc). Haven't run into any performance problems. – Exhale

There's a new option in Java 7, a ZipFileSystem – Stradivari

It must be noted that TrueVFS, the successor of TrueZIP, uses Java 7 NIO 2 features under the hood when appropriate but offers much more features like thread-safe async parallel compression. – Gilligan

Added TrueVFS code samples in separate answer. – Gilligan

Answer 4

You could use this bit of code I wrote

public static void addFilesToZip(File source, File[] files)
{
    try
    {

        File tmpZip = File.createTempFile(source.getName(), null);
        tmpZip.delete();
        if(!source.renameTo(tmpZip))
        {
            throw new Exception("Could not make temp file (" + source.getName() + ")");
        }
        byte[] buffer = new byte[1024];
        ZipInputStream zin = new ZipInputStream(new FileInputStream(tmpZip));
        ZipOutputStream out = new ZipOutputStream(new FileOutputStream(source));

        for(int i = 0; i < files.length; i++)
        {
            InputStream in = new FileInputStream(files[i]);
            out.putNextEntry(new ZipEntry(files[i].getName()));
            for(int read = in.read(buffer); read > -1; read = in.read(buffer))
            {
                out.write(buffer, 0, read);
            }
            out.closeEntry();
            in.close();
        }

        for(ZipEntry ze = zin.getNextEntry(); ze != null; ze = zin.getNextEntry())
        {
            out.putNextEntry(ze);
            for(int read = zin.read(buffer); read > -1; read = zin.read(buffer))
            {
                out.write(buffer, 0, read);
            }
            out.closeEntry();
        }

        out.close();
        tmpZip.delete();
    }
    catch(Exception e)
    {
        e.printStackTrace();
    }
}

Narayan

And with this code the new files have top priority over the old ones – Narayan

you can also change the buffer size to need, the one that is in the code right now is only for small files – Narayan

really liked this code but i needed something else where i needed to add files into folders in the zip and not just the root of the zip i posted my edited method here #9300615 hope it helps out others thanks a ton Liam for the great base code didn't really change much but i think that's a great method now :) – Heterosis

Answer 5

I don't know of a Java library that does what you describe. But what you described is practical. You can do it in .NET, using DotNetZip.

Michael Krauklis is correct that you cannot simply "append" data to a war file or zip file, but it is not because there is an "end of file" indication, strictly speaking, in a war file. It is because the war (zip) format includes a directory, which is normally present at the end of the file, that contains metadata for the various entries in the war file. Naively appending to a war file results in no update to the directory, and so you just have a war file with junk appended to it.

What's necessary is an intelligent class that understands the format, and can read+update a war file or zip file, including the directory as appropriate. DotNetZip does this, without uncompressing/recompressing the unchanged entries, just as you described or desired.

Oakland

Answer 6

As Cheeso says, there's no way of doing it. AFAIK the zip front-ends are doing exactly the same as you internally.

Anyway if you're worried about the speed of extracting/compressing everything, you may want to try the SevenZipJBindings library.

I covered this library in my blog some months ago (sorry for the auto-promotion). Just as an example, extracting a 104MB zip file using the java.util.zip took me 12 seconds, while using this library took 4 seconds.

In both links you can find examples about how to use it.

Hope it helps.

Baudelaire

@carlos regarding your blog post: which Java version did you use? I just tested getting size of a 148M ZIP archive with standard API (new ZipFile(file).size()) and latest 7Zip bindings with Java 1.6.0_17 on a amd64 Linux system (4 cores). The standard API outperformed 7Zip by far (at least for the task you present on your blog: getting number of entries). Java took an avg of 1.5ms while 7Zip needed an avg of 350ms for 100 runs (excluding warmup). So from my perspective, there is no need to throw native libraries at this kind of problem. – Tigress

Didn't realise that this was going to use a native library thanks for point that out - will not investigate further. – Aeneid

@Carlos: If you have some free time, can you compare extraction to Apache common compress (commons.apache.org/compress)? – Burdett

@dma_k: I could do the test but the documentation says 'gzip support is provided by the java.util.zip package of the Java class library.' So I don't expect any difference – Baudelaire

I confirm that (after checking commons-compress sources): it utilizes available algorithms where possible. They have created their own ZipFile implementation, but it is based on java.util.zip.Inflater et al. I don't expect any tremendous speed boost as well, but comparison of extraction from .zip file might be interesing for you just for completeness. – Burdett

Answer 7

See this bug report.

Using append mode on any kind of structured data like zip files or tar files is not something you can really expect to work. These file formats have an intrinsic "end of file" indication built into the data format.

If you really want to skip the intermediate step of un-waring/re-waring, you could read the war file file, get all the zip entries, then write to a new war file "appending" the new entries you wanted to add. Not perfect, but at least a more automated solution.

Wilkey

I am not sure how your proposed solution differs from what I am doing already - how is this more automated? – Aeneid

I am still keen to understand your solution - you say instead or un-war then re-war I should read the file and then write to a new war - is this not the same thing? Please can you explain – Aeneid

Answer 8

Yet Another Solution: You may find code below useful in other situations as well. I have used ant this way to compile Java directories, generating jar files, updating zip files,...

    public static void antUpdateZip(String zipFilePath, String libsToAddDir) {
    Project p = new Project();
    p.init();

    Target target = new Target();
    target.setName("zip");
    Zip task = new Zip();
    task.init();
    task.setDestFile(new File(zipFilePath));
    ZipFileSet zipFileSet = new ZipFileSet();
    zipFileSet.setPrefix("WEB-INF/lib");
    zipFileSet.setDir(new File(libsToAddDir));
    task.addFileset(zipFileSet);
    task.setUpdate(true);

    task.setProject(p);
    task.init();
    target.addTask(task);
    target.setProject(p);
    p.addTarget(target);

    DefaultLogger consoleLogger = new DefaultLogger();
    consoleLogger.setErrorPrintStream(System.err);
    consoleLogger.setOutputPrintStream(System.out);
    consoleLogger.setMessageOutputLevel(Project.MSG_DEBUG);
    p.addBuildListener(consoleLogger);

    try {
        // p.fireBuildStarted();

        // ProjectHelper helper = ProjectHelper.getProjectHelper();
        // p.addReference("ant.projectHelper", helper);
        // helper.parse(p, buildFile);
        p.executeTarget(target.getName());
        // p.fireBuildFinished(null);
    } catch (BuildException e) {
        p.fireBuildFinished(e);
        throw new AssertionError(e);
    }
}

Rema

Answer 9

this a simple code to get a response with using servlet and send a response

myZipPath = bla bla...
    byte[] buf = new byte[8192];
    String zipName = "myZip.zip";
    String zipPath = myzippath+ File.separator+"pdf" + File.separator+ zipName;
    File pdfFile = new File("myPdf.pdf");
    ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipPath));
    ZipEntry zipEntry = new ZipEntry(pdfFile.getName());
    out.putNextEntry(zipEntry);
    InputStream in = new FileInputStream(pdfFile);
    int len;
    while ((len = in.read(buf)) > 0) {
         out.write(buf, 0, len);
     }
    out.closeEntry();
    in.close();
     out.close();
                FileInputStream fis = new FileInputStream(zipPath);
                response.setContentType("application/zip");
                response.addHeader("content-disposition", "attachment;filename=" + zipName);
    OutputStream os = response.getOutputStream();
            int length = is.read(buffer);
            while (length != -1)
            {
                os.write(buffer, 0, length);
                length = is.read(buffer);
            }

Gasaway

Answer 10

Here are examples how easily files can be appended to existing zip using TrueVFS:

// append a file to archive under different name
TFile.cp(new File("existingFile.txt"), new TFile("archive.zip", "entry.txt"));

// recusively append a dir to the root of archive
TFile src = new TFile("dirPath", "dirName");
src.cp_r(new TFile("archive.zip", src.getName()));

TrueVFS, the successor of TrueZIP, uses Java 7 NIO 2 features under the hood when appropriate but offers much more features like thread-safe async parallel compression.

Beware also that Java 7 ZipFileSystem by default is vulnerable to OutOfMemoryError on huge inputs.

Gilligan

Answer 11

Here is Java 1.7 version of Liam answer which uses try with resources and Apache Commons IO.

The output is written to a new zip file but it can be easily modified to write to the original file.

  /**
   * Modifies, adds or deletes file(s) from a existing zip file.
   *
   * @param zipFile the original zip file
   * @param newZipFile the destination zip file
   * @param filesToAddOrOverwrite the names of the files to add or modify from the original file
   * @param filesToAddOrOverwriteInputStreams the input streams containing the content of the files
   * to add or modify from the original file
   * @param filesToDelete the names of the files to delete from the original file
   * @throws IOException if the new file could not be written
   */
  public static void modifyZipFile(File zipFile,
      File newZipFile,
      String[] filesToAddOrOverwrite,
      InputStream[] filesToAddOrOverwriteInputStreams,
      String[] filesToDelete) throws IOException {


    try (ZipOutputStream out = new ZipOutputStream(new FileOutputStream(newZipFile))) {

      // add existing ZIP entry to output stream
      try (ZipInputStream zin = new ZipInputStream(new FileInputStream(zipFile))) {
        ZipEntry entry = null;
        while ((entry = zin.getNextEntry()) != null) {
          String name = entry.getName();

          // check if the file should be deleted
          if (filesToDelete != null) {
            boolean ignoreFile = false;
            for (String fileToDelete : filesToDelete) {
              if (name.equalsIgnoreCase(fileToDelete)) {
                ignoreFile = true;
                break;
              }
            }
            if (ignoreFile) {
              continue;
            }
          }

          // check if the file should be kept as it is
          boolean keepFileUnchanged = true;
          if (filesToAddOrOverwrite != null) {
            for (String fileToAddOrOverwrite : filesToAddOrOverwrite) {
              if (name.equalsIgnoreCase(fileToAddOrOverwrite)) {
                keepFileUnchanged = false;
              }
            }
          }

          if (keepFileUnchanged) {
            // copy the file as it is
            out.putNextEntry(new ZipEntry(name));
            IOUtils.copy(zin, out);
          }
        }
      }

      // add the modified or added files to the zip file
      if (filesToAddOrOverwrite != null) {
        for (int i = 0; i < filesToAddOrOverwrite.length; i++) {
          String fileToAddOrOverwrite = filesToAddOrOverwrite[i];
          try (InputStream in = filesToAddOrOverwriteInputStreams[i]) {
            out.putNextEntry(new ZipEntry(fileToAddOrOverwrite));
            IOUtils.copy(in, out);
            out.closeEntry();
          }
        }
      }

    }

  }

Footrest

Answer 12

this works 100% , if you dont want to use extra libs .. 1) first, the class that append files to the zip ..

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class AddZip {

    public void AddZip() {
    }

    public void addToZipFile(ZipOutputStream zos, String nombreFileAnadir, String nombreDentroZip) {
        FileInputStream fis = null;
        try {
            if (!new File(nombreFileAnadir).exists()) {//NO EXISTE 
                System.out.println(" No existe el archivo :  " + nombreFileAnadir);return;
            }
            File file = new File(nombreFileAnadir);
            System.out.println(" Generando el archivo '" + nombreFileAnadir + "' al ZIP ");
            fis = new FileInputStream(file);
            ZipEntry zipEntry = new ZipEntry(nombreDentroZip);
            zos.putNextEntry(zipEntry);
            byte[] bytes = new byte[1024];
            int length;
            while ((length = fis.read(bytes)) >= 0) {zos.write(bytes, 0, length);}
            zos.closeEntry();
            fis.close();

        } catch (FileNotFoundException ex ) {
            Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
        } 
    }

}

2) you can call it in your controller ..

//in the top
try {
fos = new FileOutputStream(rutaZip);
zos =   new ZipOutputStream(fos);
} catch (FileNotFoundException ex) {
Logger.getLogger(UtilZip.class.getName()).log(Level.SEVERE, null, ex);
}

...
//inside your method
addZip.addToZipFile(zos, pathFolderFileSystemHD() + itemFoto.getNombre(), "foto/" + itemFoto.getNombre());

Amary

Answer 13

Based on the answer given by @sfussenegger above, following code is used to append to a jar file and download it:

public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

    Resource resourceFile = resourceLoader.getResource("WEB-INF/lib/custom.jar");
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    try (ZipOutputStream zos = new ZipOutputStream(baos, StandardCharsets.ISO_8859_1);) {
        try (ZipFile zin = new ZipFile(resourceFile.getFile(), StandardCharsets.ISO_8859_1);) {
            zin.stream().forEach((entry) -> {
                try {
                    zos.putNextEntry(entry);
                    if (!entry.isDirectory()) {
                        zin.getInputStream(entry).transferTo(zos);
                    }
                    zos.closeEntry();
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
            });
        }
        /* build file records to be appended */
        ....
        for (FileContents record : records) {
            zos.putNextEntry(new ZipEntry(record.getFileName()));
            zos.write(record.getBytes());
            zos.closeEntry();
        }
        zos.flush();
    }

    response.setContentType("application/java-archive");
    response.setContentLength(baos.size());
    response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"custom.jar\"");
    try (BufferedOutputStream out = new BufferedOutputStream(response.getOutputStream())) {
        baos.writeTo(out);
    }
}

Frenchpolish

Answer 14

zipFolder = reports is folder where all the files will be presented and this will compressed. Location of zipFolder = "C:\JavaProgrammingPractice\MavenHelloWorld\bld_root_9\reports.zip"; readfilename: This is the file and its contents of will be written in zipFolder. The file locations is same as zipFolder location. Location of readfilename = "C:\JavaProgrammingPractice\MavenHelloWorld\bld_root_9\reports\readfilename"; fs.getPath(readfilename) = will show the absolute path of the file. nf.getFileName() = Only filename is required as argument of newBufferedWriter.

public static void addFileToZipFolder(String zipFolder, String readfilename) throws IOException {
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Path.of(zipFolder);
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env))
{
  Path nf = fs.getPath(readfilename);
  String text = Files.readString(Path.of(readfilename));
  try (Writer writer = Files.newBufferedWriter(nf.getFileName(), StandardCharsets.UTF_8, StandardOpenOption.CREATE)) {
    writer.write(text);
  }
}

}

Choriocarcinoma