gradle - download and unzip file from url

Asked 11/4, 2014 at 21:40 Answered 3/2, 2020 at 16:37

What would be the proper gradle way of downloading and unzipping the file from url (http)?

If possible, I'd like to prevent re-downloading each time I run the task (in ant.get can be achieved by skipexisting: 'true').

My current solution would be:

task foo {
  ant.get(src: 'http://.../file.zip', dest: 'somedir', skipexisting: 'true')
  ant.unzip(src: 'somedir' + '/file.zip', dest: 'unpackdir')
}

still, I'd expect ant-free solution. Any chance to achieve that?

Cephalo answered 11/4, 2014 at 21:40 Comment(3)

Don't forget to wrap the execution part of a task with doLast { ... } (same mistake as in your previous question). – Jacki 11/4, 2014 at 22:20

~~BOUNTY~~ Can anyone provide an example for the answer below: "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ? – Hyperdulia 16/12, 2015 at 21:22

netflix has released some plugin: github.com/nebula-plugins/nebula-core - not sure, why this is not maintained anymore. – Martinson 19/8, 2019 at 20:12

There isn't currently a Gradle API for downloading from a URL. You can implement this using Ant, Groovy, or, if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL. The unzipping can be done in the usual Gradle way (copy method or Copy task).

Jacki answered 11/4, 2014 at 22:20 Comment(5)

I would appreciate an example... I'm a total gradle newbie! – Heterodox 5/3, 2015 at 14:12

~~BOUNTY~~ Can anyone provide an example for this answer "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ? – Hyperdulia 16/12, 2015 at 21:23

@Hyperdulia why you no groovy? – Tabulate 17/12, 2015 at 5:20

@Tabulate my current solution is using groovy and it works fine, but I wanted one using repository to benefit from the caching system. – Hyperdulia 17/12, 2015 at 5:41

@Hyperdulia Ah, caching! Posted an answer below. – Tabulate 17/12, 2015 at 5:42

Let's say you want to download this zip file as a dependency:

https://github.com/jmeter-gradle-plugin/jmeter-gradle-plugin/archive/1.0.3.zip

You define your ivy repo as:

repositories {
    ivy {
        url 'https://github.com/'

        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }

        // This is required in Gradle 6.0+ as metadata file (ivy.xml) 
        // is mandatory. Docs linked below this code section
        metadataSources { artifact() } 
    }
}

reference for required metadata here

The dependency can then be used as:

dependencies {
    compile 'jmeter-gradle-plugin:jmeter-gradle-plugin:1.0.3@zip'
    //This maps to the pattern: [organisation]:[module]:[revision]:[classifier]@[ext]         
}

To unzip:

task unzip(type: Copy) {

  def zipPath = project.configurations.compile.find {it.name.startsWith("jmeter") }
  println zipPath
  def zipFile = file(zipPath)
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir

}

optional:

If you have more than one repository in your project, it may also help (for build time and somewhat security) to restrict dependency search with relevant repositories.

Gradle 6.2+:

repositories {
    mavenCentral()
    def github = ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
    }
    exclusiveContent {
        forRepositories(github)
        filter { includeGroup("jmeter-gradle-plugin") }
    }
}

Earlier gradle versions:

repositories {
    mavenCentral {
        content { excludeGroup("jmeter-gradle-plugin") }
    }
    ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
        content { includeGroup("jmeter-gradle-plugin") }
    }
}

Tabulate answered 17/12, 2015 at 5:24 Comment(16)

@Tabulate how does 1.0.3:class@zip get matched to [revision].[ext] ? – Selfcontent 27/1, 2016 at 5:49

@Selfcontent class in that string is a classifier. It is optional, and here it isn't being used in the ivy url at all. 1.0.3 is revision and zip is ext. – Tabulate 27/1, 2016 at 12:59

Is this for android or java? I'm trying to use it but it seems like "project.configuration.compile" does not exist? I've tried to add apply java before in the script, but that does not make a difference. Can't find it in the documentation either. Using com.android.tools.build:gradle:2.1.0 and version 2.10 of gradle. – Joyajoyan 11/5, 2016 at 13:12

@Joyajoyan it should be project.configurations, mind the s in configurations. The android plugin implicitly adds the java plugin to a project. It shouldn't matter if you add it explicitly again. – Tabulate 11/5, 2016 at 13:51

It was a spelling mistake by me. When I write project.configurations and let the IDE autofill the possibilites the compile isn't one of them. And then I get a build error. Commenting out the line makes it build again. – Joyajoyan 11/5, 2016 at 13:57

please raise a new question, and refer to this one. – Tabulate 11/5, 2016 at 15:33

Unfortunately URLs with query strings cannot be accessed this way, '?' is encoded by github.com/gradle/gradle/blob/master/subprojects/resources/src/… – Wreckful 21/4, 2017 at 18:11

When I am trying this rather than looking for .zip file it is looking for a .xml file. Any way around for this? I followed exactly the same instruction. – Marque 26/6, 2018 at 10:9

Doesn't work for me. Gradle is looking for an ivy.xml file in addition to the artifact. – Antenatal 29/6, 2018 at 15:42

Does work, as others have stated, Gradle looks for .xml file. – Daveen 1/10, 2020 at 2:7

This is a great answer. I would only add that the dependency should probably be compileOnly to prevent the raw zip ending up in the runtime artifacts. – Abeabeam 19/10, 2020 at 9:29

gradle 7 now changing configurations - 'compile' removed. adding a configuration in replacement or updating to the new names still worked. – Jaffe 12/8, 2021 at 10:48

Doesn't work. Gradle 7.2 is looking for a normal maven dependency. – Metchnikoff 7/9, 2021 at 13:17

@Metchnikoff I'd expect gradle to look in all available repositories for all dependencies. If you want to restrict certain deps to certain repos, see the optional section in the answer – Tabulate 7/9, 2021 at 15:0

I can confirm this works with 7.2, @Metchnikoff have you resolved? – Combustion 21/10, 2021 at 16:2

@Combustion I solved it in another way. Because I spent a lot of time trying to work with this approach. Community on Gradle's Slack said a better way to handle my original issue. But thanks for confirming that's working. Here Gradle wasn't recognizing the pattern "/[organisation]/[module]/archive/[revision].[ext]" on dependencies. – Metchnikoff 21/10, 2021 at 18:49

    plugins {
        id 'de.undercouch.download' version '4.0.0'
    }

    /**
     * The following two tasks download a ZIP file and extract its
     * contents to the build directory
     */
    task downloadZipFile(type: Download) {
        src 'https://github.com/gradle-download-task/archive/1.0.zip'
        dest new File(buildDir, '1.0.zip')
    }

    task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
        from zipTree(downloadZipFile.dest)
        into buildDir
    }

https://github.com/michel-kraemer/gradle-download-task

Blamable answered 19/8, 2019 at 11:10 Comment(1)

This is not really a good solution compared to Ant as it is really slow. 26 minutes download instead of 2 minutes with Chrome. – Classmate 29/10, 2019 at 14:30

Jacki answered 11/4, 2014 at 22:20 Comment(5)

I would appreciate an example... I'm a total gradle newbie! – Heterodox 5/3, 2015 at 14:12

@Hyperdulia why you no groovy? – Tabulate 17/12, 2015 at 5:20

@Tabulate my current solution is using groovy and it works fine, but I wanted one using repository to benefit from the caching system. – Hyperdulia 17/12, 2015 at 5:41

@Hyperdulia Ah, caching! Posted an answer below. – Tabulate 17/12, 2015 at 5:42

Unzipping using the copy task works like this:

task unzip(type: Copy) {
  def zipFile = file('src/dists/dist.zip')
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir
}

http://mrhaki.blogspot.de/2012/06/gradle-goodness-unpacking-archive.html

Giorgia answered 30/10, 2014 at 13:48 Comment(2)

I tried moving the from zipTree() & into outputDir to the execution phase by moving them to doLast { }. What I did not understand was that it never worked. Even the println code in doLast did not work. However, this code works in the gradle configuration phase. Why? – Pulling 20/12, 2018 at 9:18

Because it configures the task. The execution still is done during execution phase. Changing the configuration during the execution phase should not be done actually and in your case the execution phase would not be executed at all, as the task thinks there is nothing to copy, so the task execution is skipped. But actually this answer is not related to the question anyway, as this does not work with an HTTP URL. – Dither 29/10, 2019 at 15:26

This works with Gradle 5 (tested with 5.5.1):

task download {
    doLast {
        def f = new File('file_path')
        new URL('url').withInputStream{ i -> f.withOutputStream{ it << i }}
    }
}

Calling gradle download downloads the file from url to file_path.

You can use the other methods from other answers to unzip the file if necessary.

Daphne answered 30/7, 2019 at 13:12 Comment(1)

Also works in gradle 4.7. In case someone needs it. – Kuo 7/7, 2021 at 17:5

I got @RaGe's answer working, but I had to adapt it since the ivy layout method has been depreciated see https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.repositories.IvyArtifactRepository.html#org.gradle.api.artifacts.repositories.IvyArtifactRepository:layout(java.lang.String,%20groovy.lang.Closure)

So to get it working I had to adjust it to this for a tomcat keycloak adapter:

ivy {
    url 'https://downloads.jboss.org/'
    patternLayout {
        artifact '/[organization]/[revision]/adapters/keycloak-oidc/[module]-[revision].[ext]'
    }
}

dependencies {
    // https://downloads.jboss.org/keycloak/4.8.3.Final/adapters/keycloak-oidc/keycloak-tomcat8-adapter-dist-4.8.3.Final.zip
    compile "keycloak:keycloak-tomcat8-adapter-dist:$project.ext.keycloakAdapterVersion@zip"
}

task unzipKeycloak(type: Copy) {

    def zipPath = project.configurations.compile.find {it.name.startsWith("keycloak") }
    println zipPath
    def zipFile = file(zipPath)
    def outputDir = file("${buildDir}/tomcat/lib")

    from zipTree(zipFile)
    into outputDir
}

Carlie answered 22/1, 2019 at 17:23 Comment(2)

Unfortunately, this solution also has Gradle looking for an Ivy XML file. – Appreciative 17/4, 2019 at 16:51

Doesn't work. Gradle 7.2 is looking for a normal maven dependency. – Metchnikoff 7/9, 2021 at 13:16

"native gradle", only the download part (see other answers for unzipping)

task foo {
  def src = 'http://example.com/file.zip'
  def destdir = 'somedir'
  def destfile = "$destdir/file.zip"
  doLast {
    def url = new URL(src)
    def f = new File(destfile)
    if (f.exists()) {
      println "file $destfile already exists, skipping download"
    } else {
      mkdir "$destdir"
      println "Downloading $destfile from $url..."
      url.withInputStream { i -> f.withOutputStream { it << i } }
    }
  }
}

Realgar answered 3/2, 2020 at 16:37 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags