Using a large ZIP/WAR as a semi-repository of JARs for compilation and builds in IDE

[Andrew Schetinin]

Hi,

For quite some time, I'm trying to find a solution for a certain problem in
our build scripts.

We have a large 3rd party WAR file (part of Eclipse BIRT - a reporting
engine) which is provided as-is in an open-source project distribution
(unfortunately, no Maven repositories).
It contains about 150+ JARs inside, and has size of 60 MB, which makes it
practical to use it as a whole - easier to switch to a new version that way.

We want to save this WAR in our local repository, then in the build script
download it as a dependency, unpack it, use contained JARs for compilation
of our custom additions (with IDE support), and then re-package it back
with our additions for usage on our server.

I have a partial solution that allows compilation and building from the
command line, but I don't succeed to fix IDE support - at least Eclipse -
it is impossible to work with that sub-project in Eclipse since the
libraries are unresolved.

Below you may find an explanation of what we succeeded to do, and
reasoning, but before I'd like to ask several questions about possible ways
of approaching the problem:

-------------------------------------------------------------------------------------------------------------------
1. How to to define dependencies that depend on each other, with complex
logic?

Suppose I want to define a dependency that downloads that WAR file, and
another dependency that extracts JAR files and uses them (there are working
examples of defining dependencies on plain folders)

configurations {
birtFiles
}

dependencies {
birtFiles 'com.eclipse.birt:birt:***@war'

compile functionExtractBirtFiles() // this function extracts WAR and
creates a list of extracted JAR files
}

This does not work, because when I access "configurations.birtFiles" from
inside functionExtractBirtFiles(), the entire configurations collection
becomes read-only, and the returned list cannot be attached to compile
dependencies.
I did not succeed to solve that, and therefore I change compilation
classpath with compileJava.doFirst() hook, which unfortunately does not
work for eclipseClasspath
Another problem with this approach is that dependency resolution works
before clean task, and I'm extracting JAR files to build directory, and it
gets removed before compilation.
Most probably, I'm trying to bend the dependency resolution mechanism for
something it is not designed for.

-------------------------------------------------------------------------------------------------------------------
2. How to trigger Ivy dependency resolution manually for a single artifact?

This actually might be a silver bullet here.
Imagine not defining any dependency on the original WAR file, but simply
manually downloading it from our Maven repository (better reusing regular
Gradle cache).
Everything happens inside functionExtractBirtFiles() and the classpath is
created for already existing JAR files.

dependencies {
// this function donwloads and extracts WAR and creates a list of
extracted JAR files
compile functionExtractBirtFiles( 'com.eclipse.birt:birt:***@war' )
}

Again, there is a problem with placing the extracted JAR files somewhere so
that the clean task does not remove them, but that can be solved by placing
them outside build directory.

Is there an easy way to manually download a file from a Maven repository as
part of some task/function in Gradle?

-------------------------------------------------------------------------------------------------------------------
3. How to alter classpath in eclipseClasspath task?

As I explain below, I succeeded to hack a solution for regular compilation,
but not for IDE support.
Even though the solution in the item 2 would be better, it would be nice to
know if there is a solution for this task as well.

eclipseClasspath.plusConfigurations only accepts configurations, and I
cannot create a configuration for extracted JAR files, because they don't
exist on the stage when all dependencies are downloaded.

There are few useful properties/methods in eclipseClasspath. For example,
eclipseClasspath.containers is unclear what it is and how to use it. Same
about beforeConfigured clause.

-------------------------------------------------------------------------------------------------------------------
This is how it works at the moment (and the reason for not working Eclipse
support):

configurations {
birtFiles
expandedBirtFiles
}

dependencies {
birtFiles 'com.eclipse.birt:birt:***@war'
}

// Extract BIRT JARs before compiling our sources
compileJava.dependsOn birt_extract

// Add extracted JARs to the classpath
compileJava.doFirst {
setClasspath createClassPath( true )
}

// this will be executed during the configuration phase
task birt_extract {
inputs.files configurations.birtFiles
def destDir = new File( buildDir, 'extracted-birt.war' )
outputs.dir destDir // this defines the output file for this task
(success indicator)
outputs.upToDateWhen { // additional success indication
def allJars = fileTree( destDir ).matching { include '**/*.jar' }
!allJars.isEmpty() // if there are any JARs extracted from WAR
artifact
}
}

// This will be executed during the actual build phase
birt_extract << {
def destDir = outputs.files.singleFile
createSingleDir( destDir.parentFile ) // first create "build"
createSingleDir( destDir ) // now create our target subfolder
inputs.files.each { File f ->
copy {
from zipTree( f ).matching { include '**/*' } // this gets a
complete file list from WAR
into destDir
}
}
}

def createClassPath( boolean withCompile ) {
def destDir = birt_extract.outputs.files.singleFile
FileCollection cp = fileTree( libDir ).matching { include '**/*.jar' }
// the following line causes an exception, because we cannot read
"configuration"
// while inside "dependencies {}", and this method is called from
inside there.
if( withCompile ) {
cp = cp.plus( configurations.compile ) // need this plus() call,
since all file collections are read-only
}
return cp
}

Basically I define a dependency on the original WAR in a regular way so it
is downloaded, then before compilation I extract the needed files, and then
set the compilation classpath to the original classpath + all extracted
JARs.

It works fine for the compilation, but it does not work for
eclipseClasspath task, because I did not find any simple way of changing
the classpath there (see the item 3).

Thanks in advance for any useful comments and suggestions!

Regards,

Andrew