How should one structure multi-project builds?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How should one structure multi-project builds?

pbarnes
First question, relating to multi-project builds:

I have three web applications and a background application that all depend on a 'core' custom library that is basically driven by Spring and Hibernate.  Currently my directory structure is something like:

\projects\WebApp1\
\projects\WebApp2\
\projects\WebApp3\
\projects\BackgroundApp\
\projects\Core\
\projects\Web\

Where Core contains shared services and data access, and Web contains shared web framework classes and utilities.

If I understand the documentation correctly, I could have a 'gradlefile' in each of the subdirectories with dependsOn(':Core') for all the applications and additionally dependsOn(':Web') for the three web applications.

However, I'm not sure what to do with the 'gradlesettings' file.  I could "include 'Core', 'Web', 'WebApp1', 'WebApp2', 'WebApp3', 'BackgroundApp'", but I wasn't sure this was the right approach, since WebApp1,2,3 are all really independent of one another, and of course so is BackgroundApp.

Second question, related to dependencies:  I'm probably going to show my ignorance with Ivy here, but here goes.  ;)

In the above example, I can specify my dependencies fine for compile time, and the project compiles fine.  However, when I try to run "gradle dists" I run into issues with what I assume are "runtime dependencies" from my "compile dependencies" as I see non-optional Hibernate dependencies are pulled down (i.e. ehcache, asm-attrs, etc.).  But for Sun jars, such as 'jta.jar' these aren't available in the Maven repository (I assume due to license).  However, the Hibernate POM describes this dependency as required and therefore I assume Ivy tries to retrieve it, fails, and the build terminates.

Is my only recourse here to create my own clientModule with the correct dependencies or is there some way I can 'override' where this one dependency is looked up?  It doesn't seem to follow any of the classpathResolvers.add() that I specified, and always (only) looks to the Maven rep.

Thanks in advance,

Phil..
Reply | Threaded
Open this post in threaded view
|

Re: How should one structure multi-project builds?

hans_d
Administrator
Hi Phil,

sorry for not answering earlier.

On May 11, 2008, at 11:55 PM, pbarnes wrote:

First question, relating to multi-project builds:

I have three web applications and a background application that all depend
on a 'core' custom library that is basically driven by Spring and Hibernate. 
Currently my directory structure is something like:

\projects\WebApp1\
\projects\WebApp2\
\projects\WebApp3\
\projects\BackgroundApp\
\projects\Core\
\projects\Web\

Where Core contains shared services and data access, and Web contains shared
web framework classes and utilities.


You can structure it like above but you could also structure it like:

projects/Core
projects/BackgroundApp
projects/Web
projects/Web/WebAppX

The Web project could have two roles. One role is to provide shared configuration, the other is to provide a library.

In the Web gradlefile you could configure for example the subprojects like:

dependencies {
compile project(':Core')
}

subprojects {
        usePlugin('java')
        type = 'war'
dependencies {
             compile "javax:servlet-api:2.5", project(':Web'), project(':Core')
        }
}

If a subproject does not have any other specific behavior, you might not even need to provide a gradefile for it.

I don't know whether Web has a direct dependency on Core and whether the WebAppX projects have such a direct dependency. If both have a direct dependency you might do it as above. If the WebAppX does not have one, only Web itself, you would remove project(':Core') from the subproject dependencies declarations.

If I understand the documentation correctly, I could have a 'gradlefile' in
each of the subdirectories with dependsOn(':Core') for all the applications
and additionally dependsOn(':Web') for the three web applications.


A gradlefile can be in every project but is optional. If you use configuration injection you might not use one. Or you use a gradlefile to define the specific stuff for a particular project (e.g. WebApp3), and declare the common things of all WebApps in the Web project's gradlefile.

However, I'm not sure what to do with the 'gradlesettings' file.  I could
"include 'Core', 'Web', 'WebApp1', 'WebApp2', 'WebApp3', 'BackgroundApp'",
but I wasn't sure this was the right approach, since WebApp1,2,3 are all
really independent of one another, and of course so is BackgroundApp.

This is the right approach. You need to include all projects which should take part in the build independent of there relationships. 


Second question, related to dependencies:  I'm probably going to show my
ignorance with Ivy here, but here goes.  ;)

In the above example, I can specify my dependencies fine for compile time,
and the project compiles fine.  However, when I try to run "gradle dists" I
run into issues with what I assume are "runtime dependencies" from my
"compile dependencies" as I see non-optional Hibernate dependencies are
pulled down (i.e. ehcache, asm-attrs, etc.).  But for Sun jars, such as
'jta.jar' these aren't available in the Maven repository (I assume due to
license).  However, the Hibernate POM describes this dependency as required
and therefore I assume Ivy tries to retrieve it, fails, and the build
terminates.

Is my only recourse here to create my own clientModule with the correct
dependencies or is there some way I can 'override' where this one dependency
is looked up?  It doesn't seem to follow any of the classpathResolvers.add()
that I specified, and always (only) looks to the Maven rep.

The usual Ivy behavior is that the resolver that finds the pom is also responsible for retrieving the jar. As Ivy allows to assign multiple URL's to resolvers this is not really a problem. But the way Gradle integrates the MavenRepo makes life harder than necessary for your I think very common use case. I have filed an issue to make this more convenient: http://jira.codehaus.org/browse/GRADLE-91

The way you have to do this now is to add another resolver.

I have to catch my plane right now and will paste the snippet how to do this in another reply hopefully coming soon.

- Hans   


Thanks in advance,

Phil..
-- 
Sent from the gradle-user mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:




--
Hans Dockter
Gradle Project lead




Reply | Threaded
Open this post in threaded view
|

Re: How should one structure multi-project builds?

pbarnes
hdockter wrote
Hi Phil,

sorry for not answering earlier.
No problem - real life always seems to get in my way too. ;)

hdockter wrote
[...snipped to shorten length...]

> However, I'm not sure what to do with the 'gradlesettings' file.  I  
> could
> "include 'Core', 'Web', 'WebApp1', 'WebApp2', 'WebApp3',  
> 'BackgroundApp'",
> but I wasn't sure this was the right approach, since WebApp1,2,3  
> are all
> really independent of one another, and of course so is BackgroundApp.

This is the right approach. You need to include all projects which  
should take part in the build independent of there relationships.
Ok, so I'm guessing that the 'gradlesettings' file is used to determine inter-project dependencies.  But can't this already be inferred from the "dependsOn()" methods?  Say 'WebApp1' depeondsOn(':Web'), and 'Web' dependsOn(':Core'), what exactly is the 'gradlesettings' file adding to the process?

And in my case, I *usually* only want to build WebApp1 OR WebApp2 OR WebApp3 - not all 3 at once.  It just so happens that all 3 dependsOn(':Core'), so they're all at the same 'root project' level.

hdockter wrote
>
> Second question, related to dependencies:  I'm probably going to  
> show my
> ignorance with Ivy here, but here goes.  ;)
>
> In the above example, I can specify my dependencies fine for  
> compile time,
> and the project compiles fine.  However, when I try to run "gradle  
> dists" I
> run into issues with what I assume are "runtime dependencies" from my
> "compile dependencies" as I see non-optional Hibernate dependencies  
> are
> pulled down (i.e. ehcache, asm-attrs, etc.).  But for Sun jars,  
> such as
> 'jta.jar' these aren't available in the Maven repository (I assume  
> due to
> license).  However, the Hibernate POM describes this dependency as  
> required
> and therefore I assume Ivy tries to retrieve it, fails, and the build
> terminates.
>
> Is my only recourse here to create my own clientModule with the  
> correct
> dependencies or is there some way I can 'override' where this one  
> dependency
> is looked up?  It doesn't seem to follow any of the  
> classpathResolvers.add()
> that I specified, and always (only) looks to the Maven rep.

The usual Ivy behavior is that the resolver that finds the pom is  
also responsible for retrieving the jar. As Ivy allows to assign  
multiple URL's to resolvers this is not really a problem. But the way  
Gradle integrates the MavenRepo makes life harder than necessary for  
your I think very common use case. I have filed an issue to make this  
more convenient: http://jira.codehaus.org/browse/GRADLE-91

The way you have to do this now is to add another resolver.
Yeah, I was assuming this was the way to do it.  But are there different resolvers for compile time versus run time?  Because, as noted in my original message, I added new resolvers via classpathResolvers.add(), but these are only 'checked' by Ivy at compile time and are not respected when resolving runtime dependencies (i.e., I don't see any attempt to use the resolvers when it's trying to resolve Hibernate's runtime dependencies).

hdockter wrote
I have to catch my plane right now and will paste the snippet how to  
do this in another reply hopefully coming soon.
- Hans
--
Hans Dockter
Gradle Project lead
http://www.gradle.org
Thanks Hans - I appreciate your help and look forward to your snippet. ;)

Phil..
Reply | Threaded
Open this post in threaded view
|

Re: How should one structure multi-project builds?

hans_d
Administrator


On May 13, 2008, at 4:16 PM, pbarnes wrote:

>
>
> hdockter wrote:
>>
>> Hi Phil,
>>
>> sorry for not answering earlier.
>>
>
> No problem - real life always seems to get in my way too. ;)
>
>
> hdockter wrote:
>>
>> [...snipped to shorten length...]
>>
>>> However, I'm not sure what to do with the 'gradlesettings' file.  I
>>> could
>>> "include 'Core', 'Web', 'WebApp1', 'WebApp2', 'WebApp3',
>>> 'BackgroundApp'",
>>> but I wasn't sure this was the right approach, since WebApp1,2,3
>>> are all
>>> really independent of one another, and of course so is  
>>> BackgroundApp.
>>
>> This is the right approach. You need to include all projects which
>> should take part in the build independent of there relationships.
>>
>
> Ok, so I'm guessing that the 'gradlesettings' file is used to  
> determine
> inter-project dependencies.  But can't this already be inferred  
> from the
> "dependsOn()" methods?  Say 'WebApp1' depeondsOn(':Web'), and 'Web'
> dependsOn(':Core'), what exactly is the 'gradlesettings' file  
> adding to the
> process?

The gradlesettings file is to tell Gradle which projects take part in  
the build. Gradle then creates project objects for those projects.  
After this the gradlefiles are executed to configure those objects.  
And each gradlefile can configure all project objects. The dependsOn  
relationship is for defining the execution order. This is a thing  
that might take a while to get used to.  In chapter 12 of the user's  
guide we describe this in more detail.

>
> And in my case, I *usually* only want to build WebApp1 OR WebApp2  
> OR WebApp3
> - not all 3 at once.  It just so happens that all 3 dependsOn
> (':Core'), so
> they're all at the same 'root project' level.
>

Yet Gradle has to evaluate all gradlefiles in any case, as the  
gradlefile of the root project or any other project might inject  
behavior into WebApp1 for example. At execution time you might want  
to build only WebApp1, but for this all projects need to be evaluated.

>
> hdockter wrote:
>>
>>>
>>> Second question, related to dependencies:  I'm probably going to
>>> show my
>>> ignorance with Ivy here, but here goes.  ;)
>>>
>>> In the above example, I can specify my dependencies fine for
>>> compile time,
>>> and the project compiles fine.  However, when I try to run "gradle
>>> dists" I
>>> run into issues with what I assume are "runtime dependencies"  
>>> from my
>>> "compile dependencies" as I see non-optional Hibernate dependencies
>>> are
>>> pulled down (i.e. ehcache, asm-attrs, etc.).  But for Sun jars,
>>> such as
>>> 'jta.jar' these aren't available in the Maven repository (I assume
>>> due to
>>> license).  However, the Hibernate POM describes this dependency as
>>> required
>>> and therefore I assume Ivy tries to retrieve it, fails, and the  
>>> build
>>> terminates.
>>>
>>> Is my only recourse here to create my own clientModule with the
>>> correct
>>> dependencies or is there some way I can 'override' where this one
>>> dependency
>>> is looked up?  It doesn't seem to follow any of the
>>> classpathResolvers.add()
>>> that I specified, and always (only) looks to the Maven rep.
>>
>> The usual Ivy behavior is that the resolver that finds the pom is
>> also responsible for retrieving the jar. As Ivy allows to assign
>> multiple URL's to resolvers this is not really a problem. But the way
>> Gradle integrates the MavenRepo makes life harder than necessary for
>> your I think very common use case. I have filed an issue to make this
>> more convenient: http://jira.codehaus.org/browse/GRADLE-91
>>
>> The way you have to do this now is to add another resolver.
>>
>
> Yeah, I was assuming this was the way to do it.  But are there  
> different
> resolvers for compile time versus run time?  Because, as noted in my
> original message, I added new resolvers via classpathResolvers.add
> (), but
> these are only 'checked' by Ivy at compile time and are not  
> respected when
> resolving runtime dependencies (i.e., I don't see any attempt to  
> use the
> resolvers when it's trying to resolve Hibernate's runtime  
> dependencies).
>
>
> hdockter wrote:
>>
>> I have to catch my plane right now and will paste the snippet how to
>> do this in another reply hopefully coming soon.
>> - Hans
>> --
>> Hans Dockter
>> Gradle Project lead
>> http://www.gradle.org
>>
>
> Thanks Hans - I appreciate your help and look forward to your  
> snippet. ;)

Here we go.

For 0.1.5 Gradle will offer something like:

dependencies {
    addMavenRepo('url1', 'url2', ...)
}

Those url's are for additionally looking for jar's. The pom is always  
looked up in the MavenRepo but jars are looked up in the MavenRepo  
plus at the other url's.

For now I can only offer you a rather monstrous snippet:

import org.apache.ivy.plugins.resolver.URLResolver
import org.apache.ivy.plugins.resolver.DualResolver
import org.apache.ivy.plugins.resolver.IBiblioResolver

def iBiblioResolver = new IBiblioResolver()
iBiblioResolver.setUsepoms(true)
iBiblioResolver.name = name
iBiblioResolver.root = "http://repo1.maven.org/maven2/"
iBiblioResolver.pattern = "[organisation]/[module]/[revision]/
[artifact]-[revision].[ext]"
iBiblioResolver.m2compatible = true

def urlResolver = new URLResolver()
urlResolver.setM2compatible true
urlResolver.addArtifactPattern("http://www.phil-repo.com/ 
[organisation]/[module]/[revision]/[artifact]-[revision].[ext]")

DualResolver dualResolver = new DualResolver()
dualResolver.add iBiblioResolver
dualResolver.add urlResolver

dependencies {
     classpathResolvers.add(dualResolver)
     ....
}

In the urlResolver goes your other repository URL

- Hans


>
> Phil..
> --
> View this message in context: http://www.nabble.com/How-should-one- 
> structure-multi-project-builds--tp17178473p17209593.html
> Sent from the gradle-user mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

--
Hans Dockter
Gradle Project lead
http://www.gradle.org



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: How should one structure multi-project builds?

hans_d
Administrator

On May 14, 2008, at 5:26 PM, Hans Dockter wrote:

>
>
> On May 13, 2008, at 4:16 PM, pbarnes wrote:
>>
>>
>> hdockter wrote:
>>>
>>> Hi Phil,
>>>
>>> sorry for not answering earlier.
>>>
>>
>> No problem - real life always seems to get in my way too. ;)
>>
>>
>> hdockter wrote:
>>>
>>> [...snipped to shorten length...]
>>>
>>>> However, I'm not sure what to do with the 'gradlesettings' file.  I
>>>> could
>>>> "include 'Core', 'Web', 'WebApp1', 'WebApp2', 'WebApp3',
>>>> 'BackgroundApp'",
>>>> but I wasn't sure this was the right approach, since WebApp1,2,3
>>>> are all
>>>> really independent of one another, and of course so is  
>>>> BackgroundApp.
>>>
>>> This is the right approach. You need to include all projects which
>>> should take part in the build independent of there relationships.
>>>
>>
>> Ok, so I'm guessing that the 'gradlesettings' file is used to  
>> determine
>> inter-project dependencies.  But can't this already be inferred  
>> from the
>> "dependsOn()" methods?  Say 'WebApp1' depeondsOn(':Web'), and 'Web'
>> dependsOn(':Core'), what exactly is the 'gradlesettings' file  
>> adding to the
>> process?
>
> The gradlesettings file is to tell Gradle which projects take part  
> in the build. Gradle then creates project objects for those  
> projects. After this the gradlefiles are executed to configure  
> those objects. And each gradlefile can configure all project  
> objects. The dependsOn relationship is for defining the execution  
> order. This is a thing that might take a while to get used to.  In  
> chapter 12 of the user's guide we describe this in more detail.
>
>>
>> And in my case, I *usually* only want to build WebApp1 OR WebApp2  
>> OR WebApp3
>> - not all 3 at once.  It just so happens that all 3 dependsOn
>> (':Core'), so
>> they're all at the same 'root project' level.
>>
>
> Yet Gradle has to evaluate all gradlefiles in any case, as the  
> gradlefile of the root project or any other project might inject  
> behavior into WebApp1 for example. At execution time you might want  
> to build only WebApp1, but for this all projects need to be evaluated.
>
>>
>> hdockter wrote:
>>>
>>>>
>>>> Second question, related to dependencies:  I'm probably going to
>>>> show my
>>>> ignorance with Ivy here, but here goes.  ;)
>>>>
>>>> In the above example, I can specify my dependencies fine for
>>>> compile time,
>>>> and the project compiles fine.  However, when I try to run "gradle
>>>> dists" I
>>>> run into issues with what I assume are "runtime dependencies"  
>>>> from my
>>>> "compile dependencies" as I see non-optional Hibernate dependencies
>>>> are
>>>> pulled down (i.e. ehcache, asm-attrs, etc.).  But for Sun jars,
>>>> such as
>>>> 'jta.jar' these aren't available in the Maven repository (I assume
>>>> due to
>>>> license).  However, the Hibernate POM describes this dependency as
>>>> required
>>>> and therefore I assume Ivy tries to retrieve it, fails, and the  
>>>> build
>>>> terminates.
>>>>
>>>> Is my only recourse here to create my own clientModule with the
>>>> correct
>>>> dependencies or is there some way I can 'override' where this one
>>>> dependency
>>>> is looked up?  It doesn't seem to follow any of the
>>>> classpathResolvers.add()
>>>> that I specified, and always (only) looks to the Maven rep.
>>>
>>> The usual Ivy behavior is that the resolver that finds the pom is
>>> also responsible for retrieving the jar. As Ivy allows to assign
>>> multiple URL's to resolvers this is not really a problem. But the  
>>> way
>>> Gradle integrates the MavenRepo makes life harder than necessary for
>>> your I think very common use case. I have filed an issue to make  
>>> this
>>> more convenient: http://jira.codehaus.org/browse/GRADLE-91
>>>
>>> The way you have to do this now is to add another resolver.
>>>
>>
>> Yeah, I was assuming this was the way to do it.  But are there  
>> different
>> resolvers for compile time versus run time?  Because, as noted in my
>> original message, I added new resolvers via classpathResolvers.add
>> (), but
>> these are only 'checked' by Ivy at compile time and are not  
>> respected when
>> resolving runtime dependencies (i.e., I don't see any attempt to  
>> use the
>> resolvers when it's trying to resolve Hibernate's runtime  
>> dependencies).
>>
>>
>> hdockter wrote:
>>>
>>> I have to catch my plane right now and will paste the snippet how to
>>> do this in another reply hopefully coming soon.
>>> - Hans
>>> --
>>> Hans Dockter
>>> Gradle Project lead
>>> http://www.gradle.org
>>>
>>
>> Thanks Hans - I appreciate your help and look forward to your  
>> snippet. ;)
>
> Here we go.
>
> For 0.1.5 Gradle will offer something like:
>
> dependencies {
>    addMavenRepo('url1', 'url2', ...)
> }
>
> Those url's are for additionally looking for jar's. The pom is  
> always looked up in the MavenRepo but jars are looked up in the  
> MavenRepo plus at the other url's.
>
> For now I can only offer you a rather monstrous snippet:
>
> import org.apache.ivy.plugins.resolver.URLResolver
> import org.apache.ivy.plugins.resolver.DualResolver
> import org.apache.ivy.plugins.resolver.IBiblioResolver
>
> def iBiblioResolver = new IBiblioResolver()
> iBiblioResolver.setUsepoms(true)
> iBiblioResolver.name = name
> iBiblioResolver.root = "http://repo1.maven.org/maven2/"
> iBiblioResolver.pattern = "[organisation]/[module]/[revision]/
> [artifact]-[revision].[ext]"
> iBiblioResolver.m2compatible = true
>
> def urlResolver = new URLResolver()
> urlResolver.setM2compatible true
> urlResolver.addArtifactPattern("http://www.phil-repo.com/ 
> [organisation]/[module]/[revision]/[artifact]-[revision].[ext]")
>
> DualResolver dualResolver = new DualResolver()
> dualResolver.add iBiblioResolver
> dualResolver.add urlResolver
>
> dependencies {
>     classpathResolvers.add(dualResolver)
>     ....
> }
>
> In the urlResolver goes your other repository URL

With Gradle 0.2 this boils down to:

dependencies {
    addMavenRepo('http://www.phil-repo.com')
}

See user's guide 12.5.2

- Hans

>
> - Hans
>
>
>>
>> Phil..
>> --
>> View this message in context: http://www.nabble.com/How-should-one- 
>> structure-multi-project-builds--tp17178473p17209593.html
>> Sent from the gradle-user mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>     http://xircles.codehaus.org/manage_email
>>
>>
>
> --
> Hans Dockter
> Gradle Project lead
> http://www.gradle.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email