10 recipes for gradle-download-task

I received a couple of requests lately concerning common problems when downloading files in Gradle build scripts using gradle-download-task. Almost all of these issues could be solved by just using gradle-download-task the right way instead of extending the plugin. I have collected a number of example build scripts for these issues over the last months and I thought it would be good idea to summarize them here.

For those of you who don’t know what gradle-download-task is: the plugin provides a Download task that displays progress information while downloading files, just like Gradle does when it fetches an artifact from a repository.

Note: the following examples have been tested with Gradle 2.1 or later and gradle-download-task 2.1.0 or later.

Recipe #1: download a single file

This is the simplest use case for gradle-download-task. The following snippet downloads a single file to the project’s build directory.

task downloadFile(type: Download) {
    src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    dest buildDir
    overwrite true
}

Alternatively, use the download extension to download a file anywhere in your build script:

task downloadFile << {
    download {
        src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
        dest buildDir
        overwrite true
    }
}

Complete example build scripts: simpleTask.gradle and simpleExtension.gradle

Recipe #2: download multiple files

If you want to download multiple files at once to the project’s build directory use the following snippet.

task downloadFiles(type: Download) {
    src([
        'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven.org/maven2/org/eclipse/jetty/jetty-server/9.1.3.v20140225/jetty-server-9.1.3.v20140225-javadoc.jar'
    ])
    dest buildDir
    overwrite true
}

Complete example build script: multipleFiles.gradle

Recipe #3: download and rename multiple files

If you want to download multiple files and specify destination file names for each of them you have two options.

Option 1 calls the download extension in a loop.

/**
 * Define files to download and destination file names
 */
ext.src = [
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD': 'config.guess',
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD': 'config.sub'
]

/**
 * Call the download extension in a for loop
 */
task downloadMultipleFiles1 << {
    for (s in src) {
        download {
            src s.key
            dest new File(buildDir, s.value)
        }
    }
}

Option 2 creates multiple Gradle tasks to achieve the same thing.

/**
 * Define files to download and destination file names
 */
ext.src = [
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD': 'config.guess',
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD': 'config.sub'
]

/**
 * Create multiple tasks
 */
task downloadMultipleFiles2

for (s in src) {
    task "downloadMultipleFiles2_${s.key.hashCode()}"(type: Download) {
        src s.key
        dest new File(buildDir, s.value)
    }
    downloadMultipleFiles2.dependsOn("downloadMultipleFiles2_${s.key.hashCode()}")
}

Complete example build script: multipleFilesRename.gradle

Recipe #4: custom HTTP headers

Download a single file and specify a custom HTTP header with the following snippet.

task downloadFile(type: Download) {
    src 'http://download.oracle.com/otn-pub/java/jdk/8-b132/jre-8-windows-x64.exe'
    dest buildDir
    header 'Cookie', 'oraclelicense=accept-securebackup-cookie'
}

This example demonstrates how to download the Oracle JDK 8 installer from a build script by automatically accepting the Oracle License.

Complete example build script: customHeader.gradle

Recipe #5: download all files from a directory

The following snippet downloads all files from a directory on a server. It makes use of Ivy’s URL lister to read the server’s directory listing.

task downloadDirectory << {
    def dir = 'http://central.maven.org/maven2/de/undercouch/gradle-download-task/1.0/'
    def urlLister = new org.apache.ivy.util.url.ApacheURLLister()
    def files = urlLister.listFiles(new URL(dir))
    download {
       src files
       dest buildDir
    }
}

From Gradle 2.0 on you need to include a build script dependency to Apache Ivy in order to make this recipe work. Put the following right at the beginning of your build script.

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'org.apache.ivy:ivy:2.3.0'
    }
}

Complete example build script: directory.gradle

Recipe #6: mirror servers

Download a single file from a mirror server with the following snippet. It first configures multiple mirror servers and then uses the first one that is working.

task downloadFile << {
    def mirrors = [
        'http://central.maven-non-existing.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven-non-existing2.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    ] as Queue
    while(true) {
        def mirror = mirrors.poll()
        try {
            download {
                src mirror
                dest buildDir
                overwrite true
            }
            break
        } catch (Exception e) {
            if (mirrors.isEmpty()) {
                throw e
            }
            logger.warn("Could not download file. Trying next mirror.")
        }
    }
}

Complete example build script: mirrors.gradle

Recipe #7: Groovy closures for src and dest

The following snippet downloads a single file. It uses Groovy closures to generate values for the src and dest properties at runtime.

/**
 * Example data and methods
 */
ext {
    downloadToBuildDir = true
}

def getMavenCentralUrl() {
    return "http://central.maven.org/maven2/"
}

def getStylesJar() {
    return "org/citationstyles/styles/1.0/styles-1.0.jar"
}

/**
 * The actual example
 */
task downloadFile(type: Download) {
    src {
        def mavenUrl = getMavenCentralUrl()
        def stylesJar = getStylesJar()
        return mavenUrl + stylesJar
    }
    dest {
        if (downloadToBuildDir) {
            return buildDir
        }
        return File.createTempDir()
    }
    overwrite true
}

Complete example build script: srcAndDestClosure.gradle

Recipe #8: temporary .part file (i.e. download and rename)

This snippet downloads a single file conditionally using a temporary file name (<filename>.part). It renames the file afterwards if the download was successful. The .part file is always overwritten if it exists (e.g. from a previous attempt to download the file).

task downloadFile << {
    def destFile = new File(buildDir, "styles-1.0.jar")
    if (!destFile.exists()) {
        def tempFile = new File(buildDir, "styles-1.0.jar.part")
        download {
            src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
            dest tempFile
            overwrite true
        }
        tempFile.renameTo(destFile)
    }
}

Complete example build script: tempRename.gradle

Recipe #9: unzip downloaded file

The following two tasks download a ZIP file and extract its contents to the project’s build directory.

task downloadZipFile(type: Download) {
    src 'https://github.com/michel-kraemer/gradle-download-task/archive/1.0.zip'
    dest new File(buildDir, '1.0.zip')
}

task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
    from zipTree(downloadZipFile.dest)
    into buildDir
}

Complete example build script: unzip.gradle

Recipe #10: verify checksum of downloaded file

The final snippet consists of two tasks that first download a file and then verify its checksum.

task downloadFile(type: Download) {
    src 'http://www.example.com/index.html'
    dest buildDir
}

task verifyFile(type: Verify, dependsOn: downloadFile) {
    src new File(buildDir, 'index.html')
    algorithm 'MD5'
    checksum '09b9c392dc1f6e914cea287cb6be34b0'
}

Complete example build script: verify.gradle

More snippets

At the time of writing this post the gradle-download-task repository contains eleven example build scripts. You can find the in the examples directory:

https://github.com/michel-kraemer/gradle-download-task/tree/master/examples

More examples will be added in the future, so make sure to check this directory too. If you have other snippets worth adding to the list then send me a pull request on GitHub or add them here to the comments. Thanks!

More information

If you want to learn more about the gradle-download-task plugin have a look at its README file or at my earlier post.