Using a large ZIP/WAR as a semi-repository of JARs for compilation and builds in IDE

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Using a large ZIP/WAR as a semi-repository of JARs for compilation and builds in IDE

Andrew Schetinin
Hi,

For quite some time, I'm trying to find a solution for a certain problem in our build scripts.

We have a large 3rd party WAR file (part of Eclipse BIRT - a reporting engine) which is provided as-is in an open-source project distribution (unfortunately, no Maven repositories).
It contains about 150+ JARs inside, and has size of 60 MB, which makes it practical to use it as a whole - easier to switch to a new version that way.

We want to save this WAR in our local repository, then in the build script download it as a dependency, unpack it, use contained JARs for compilation of our custom additions (with IDE support), and then re-package it back with our additions for usage on our server.

I have a partial solution that allows compilation and building from the command line, but I don't succeed to fix IDE support - at least Eclipse - it is impossible to work with that sub-project in Eclipse since the libraries are unresolved.

Below you may find an explanation of what we succeeded to do, and reasoning, but before I'd like to ask several questions about possible ways of approaching the problem:

-------------------------------------------------------------------------------------------------------------------
1. How to to define dependencies that depend on each other, with complex logic?

Suppose I want to define a dependency that downloads that WAR file, and another dependency that extracts JAR files and uses them (there are working examples of defining dependencies on plain folders)

configurations {
    birtFiles
}

dependencies {
    birtFiles 'com.eclipse.birt:birt:3.7.2@war'

    compile functionExtractBirtFiles() // this function extracts WAR and creates a list of extracted JAR files
}

This does not work, because when I access "configurations.birtFiles" from inside functionExtractBirtFiles(), the entire configurations collection becomes read-only, and the returned list cannot be attached to compile dependencies.
I did not succeed to solve that, and therefore I change compilation classpath with compileJava.doFirst() hook, which unfortunately does not work for eclipseClasspath
Another problem with this approach is that dependency resolution works before clean task, and I'm extracting JAR files to build directory, and it gets removed before compilation.
Most probably, I'm trying to bend the dependency resolution mechanism for something it is not designed for.

-------------------------------------------------------------------------------------------------------------------
2. How to trigger Ivy dependency resolution manually for a single artifact?

This actually might be a silver bullet here.
Imagine not defining any dependency on the original WAR file, but simply manually downloading it from our Maven repository (better reusing regular Gradle cache).
Everything happens inside functionExtractBirtFiles() and the classpath is created for already existing JAR files.

dependencies {
    // this function donwloads and extracts WAR and creates a list of extracted JAR files
    compile functionExtractBirtFiles( 'com.eclipse.birt:birt:3.7.2@war' ) 
}

Again, there is a problem with placing the extracted JAR files somewhere so that the clean task does not remove them, but that can be solved by placing them outside build directory.

Is there an easy way to manually download a file from a Maven repository as part of some task/function in Gradle?

-------------------------------------------------------------------------------------------------------------------
3. How to alter classpath in eclipseClasspath task?

As I explain below, I succeeded to hack a solution for regular compilation, but not for IDE support.
Even though the solution in the item 2 would be better, it would be nice to know if there is a solution for this task as well.

eclipseClasspath.plusConfigurations only accepts configurations, and I cannot create a configuration for extracted JAR files, because they don't exist on the stage when all dependencies are downloaded.

There are few useful properties/methods in eclipseClasspath. For example, eclipseClasspath.containers is unclear what it is and how to use it. Same about beforeConfigured clause.

-------------------------------------------------------------------------------------------------------------------
This is how it works at the moment (and the reason for not working Eclipse support):

configurations {
    birtFiles
    expandedBirtFiles
}

dependencies {
    birtFiles 'com.eclipse.birt:birt:3.7.2@war'
}

// Extract BIRT JARs before compiling our sources
compileJava.dependsOn birt_extract

// Add extracted JARs to the classpath
compileJava.doFirst {
    setClasspath createClassPath( true )
}

// this will be executed during the configuration phase
task birt_extract {
    inputs.files configurations.birtFiles
    def destDir = new File( buildDir, 'extracted-birt.war' )
    outputs.dir destDir // this defines the output file for this task (success indicator)
    outputs.upToDateWhen { // additional success indication
        def allJars = fileTree( destDir ).matching { include '**/*.jar' }
        !allJars.isEmpty() // if there are any JARs extracted from WAR artifact
    }
}

// This will be executed during the actual build phase
birt_extract << {
    def destDir = outputs.files.singleFile
    createSingleDir( destDir.parentFile ) // first create "build"
    createSingleDir( destDir ) // now create our target subfolder
    inputs.files.each { File f ->
        copy {
            from zipTree( f ).matching { include '**/*' } // this gets a complete file list from WAR
            into destDir
        }
    }
}

def createClassPath( boolean withCompile ) {
    def destDir = birt_extract.outputs.files.singleFile
    FileCollection cp = fileTree( libDir ).matching { include '**/*.jar' }
    // the following line causes an exception, because we cannot read "configuration"
    // while inside "dependencies {}", and this method is called from inside there.
    if( withCompile ) {
        cp = cp.plus( configurations.compile ) // need this plus() call, since all file collections are read-only
    }
    return cp
}

Basically I define a dependency on the original WAR in a regular way so it is downloaded, then before compilation I extract the needed files, and then set the compilation classpath to the original classpath + all extracted JARs.

It works fine for the compilation, but it does not work for eclipseClasspath task, because I did not find any simple way of changing the classpath there (see the item 3).

Thanks in advance for any useful comments and suggestions!

Regards,

Andrew

--
--
Andrew Schetinin