Executing child project tasks from the command line

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Executing child project tasks from the command line

Adam Murdoch-2
Hi,

I am currently porting a large ant based multi-project build to gradle.  
Generally, the process has been very enjoyable, and the build is looking
heaps better. Thank you for creating gradle.

One thing I miss from our ant-based build is the ability to run a
particular target for all projects or for a particular ad-hoc subset of
projects from the top-level project.  I'd like to add something like
this to gradle.  In particular, I'd like to be able to (optionally)
specify tasks by path (rather than name) on the gradle command-line.  
Something like:

gradle :child1:test :child4:test

More specifically, when a name is provided on the command-line, if it
starts with ':', it is treated as a task path, and the task with that
path is executed (along with its dependencies). Otherwise, it is treated
as it is currently, where all tasks with the given name are executed
(with dependencies).

Would it make sense to add this behaviour to gradle? If so, I'd like to
have a go at adding it.  Some questions:

- I couldn't find anything much on the web site about contributing to
the project, eg how to provide patches, which jvms are we targeting,
testing, tracking in jira, etc

- Does it make sense to keep the --non-recursive option, as the same
thing would be possible using :task notation?

- Should project default targets be interpreted the same way?


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

astubbs
+1

We need a page on the dev wiki this with information - plus some sort of architectural overview of the internals of Gradle.

Adam Murdoch-2 wrote
- I couldn't find anything much on the web site about contributing to
the project, eg how to provide patches, which jvms are we targeting,
testing, tracking in jira, etc
Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

hans_d
Administrator
In reply to this post by Adam Murdoch-2
Hi Adam,

On Jul 16, 2008, at 12:32 AM, Adam Murdoch wrote:

> Hi,
>
> I am currently porting a large ant based multi-project build to  
> gradle.  Generally, the process has been very enjoyable, and the  
> build is looking heaps better. Thank you for creating gradle.
>
> One thing I miss from our ant-based build is the ability to run a  
> particular target for all projects or for a particular ad-hoc  
> subset of projects from the top-level project.  I'd like to add  
> something like this to gradle.  In particular, I'd like to be able  
> to (optionally) specify tasks by path (rather than name) on the  
> gradle command-line.  Something like:
>
> gradle :child1:test :child4:test
>
> More specifically, when a name is provided on the command-line, if  
> it starts with ':', it is treated as a task path, and the task with  
> that path is executed (along with its dependencies). Otherwise, it  
> is treated as it is currently, where all tasks with the given name  
> are executed (with dependencies).
>
> Would it make sense to add this behaviour to gradle?

That would make a lot of sense. In fact, as soon as we support  
arbitrary multi-project layouts this is even required. Right now the  
only way to achieve what you want (i.e. partial build) is to go to  
the subproject folder and execute the build from there. This works  
only because we assume a hierarchical layout and are able to find the  
root by looking for a settings.gradle file. As soon as we can't make  
any assumptions about the layout, you either have to specify the root  
when you are in the subproject or the subproject, if you are in the  
root. The latter is what you want.

> If so, I'd like to have a go at adding it.  Some questions:
>
> - I couldn't find anything much on the web site about contributing  
> to the project, eg how to provide patches, which jvms are we  
> targeting, testing, tracking in jira, etc

Good points. I gonna put them in the developer wiki soon. But I will  
also answer some of them directly :)

Patches via Jira Issue and attachements
JVMs: 1.5 (We have started using  the retrotranslator to produce an  
additional JDK 1.4 compatible version. But there are still some  
unsolved problems. )
Testing: JUnit 4.0 + JMock 2
Tracking in Jira? Do you mean svn tracking?

We are in the midst of refactoring our core from Groovy to Java. Any  
contributions to the core should be in Java therefore.

>
> - Does it make sense to keep the --non-recursive option, as the  
> same thing would be possible using :task notation?

Good question. I think it is a very uncommon, rather experimental  
option anyway. It prevents descending which is usually not what you  
want. And, as you have said, it is a subset in functionality of your  
notation proposal. So I can't see why we should keep it.

>
> - Should project default targets be interpreted the same way?

I think so, even though I don't see any use cases right now for  
default tasks with an absolute path. But if only for the sake of  
consistency they should behave the same way.

- Hans

>
>
> Adam
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

Adam Murdoch-2


Hans Dockter wrote:

>> - I couldn't find anything much on the web site about contributing to
>> the project, eg how to provide patches, which jvms are we targeting,
>> testing, tracking in jira, etc
>
> Good points. I gonna put them in the developer wiki soon. But I will
> also answer some of them directly :)
>
> Patches via Jira Issue and attachements
> JVMs: 1.5 (We have started using  the retrotranslator to produce an
> additional JDK 1.4 compatible version. But there are still some
> unsolved problems. )
> Testing: JUnit 4.0 + JMock 2
> Tracking in Jira? Do you mean svn tracking?
No, I just meant, do we track patches using Jira issues - which it looks
like we do

>
> We are in the midst of refactoring our core from Groovy to Java. Any
> contributions to the core should be in Java therefore.

I've added a page to the developer wiki with this detail in it.  The
page needs someone to check it, as I just made up some of the details
(being a newcomer here, I don't know how things are done in gradle-land).

http://docs.codehaus.org/display/GRADLE/How+to+contribute+a+patch+to+Gradle

Some questions:

- How do you know if you've broken anything when you change stuff? (ie
which tests do I run, and how do I do so?)

- How do you decide what counts as the 'core', and therefore needs to be
implemented in java, and what does not count as 'core', and therefore
can be (must be?) implemented in groovy.

- Does anything need to be sent to the dev mailing list before/after
making a patch letting people know what and why?


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

hans_d
Administrator
Hi Adam,

On Jul 17, 2008, at 12:48 AM, Adam Murdoch wrote:

>
>
> Hans Dockter wrote:
>>> - I couldn't find anything much on the web site about  
>>> contributing to the project, eg how to provide patches, which  
>>> jvms are we targeting, testing, tracking in jira, etc
>>
>> Good points. I gonna put them in the developer wiki soon. But I  
>> will also answer some of them directly :)
>>
>> Patches via Jira Issue and attachements
>> JVMs: 1.5 (We have started using  the retrotranslator to produce  
>> an additional JDK 1.4 compatible version. But there are still some  
>> unsolved problems. )
>> Testing: JUnit 4.0 + JMock 2
>> Tracking in Jira? Do you mean svn tracking?
> No, I just meant, do we track patches using Jira issues - which it  
> looks like we do
>
>>
>> We are in the midst of refactoring our core from Groovy to Java.  
>> Any contributions to the core should be in Java therefore.
>
> I've added a page to the developer wiki with this detail in it.  
> The page needs someone to check it, as I just made up some of the  
> details (being a newcomer here, I don't know how things are done in  
> gradle-land).
>
> http://docs.codehaus.org/display/GRADLE/How+to+contribute+a+patch+to 
> +Gradle
>
> Some questions:
>
> - How do you know if you've broken anything when you change stuff?  
> (ie which tests do I run, and how do I do so?)

In src/test/groovy are all the unit tests. Please set the following  
system properties to run them:

-ea -Dgradle.home=roadToNowhere -Xmx512M

Than there are integration tests executed by the build. You can run  
them with: gradle integTest

As trunk is only buildable with a snapshot version you might use the  
gradle wrapper to build trunk, which means: ./gradlew integTest

You can also build trunk with trunk from your IDE.

>
> - How do you decide what counts as the 'core', and therefore needs  
> to be implemented in java, and what does not count as 'core', and  
> therefore can be (must be?) implemented in groovy.

The core is everything except the DSL layer. As we are in the midst  
of a refactoring from Groovy to Java this layer isn't yet  
architecturally as separated as it should be. More or less everything  
except the DefaultProject, DefaultTask and the org.gradle.groovy  
package should be in Java.

>
> - Does anything need to be sent to the dev mailing list before/
> after making a patch letting people know what and why?

Honestly, I don't know.There is no process for this yet. The whole  
approach how to coordinate and sustain team work needs to be  
established. This is very important and I'm grateful that you are  
pushing this topic.

I try to contribute some more stuff to the Wiki tomorrow.

Cheers,

- Hans

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

Adam Murdoch-2
In reply to this post by hans_d


Hans Dockter wrote:

> Hi Adam,
>
> On Jul 16, 2008, at 12:32 AM, Adam Murdoch wrote:
>> More specifically, when a name is provided on the command-line, if it
>> starts with ':', it is treated as a task path, and the task with that
>> path is executed (along with its dependencies). Otherwise, it is
>> treated as it is currently, where all tasks with the given name are
>> executed (with dependencies).
>>
>> Would it make sense to add this behaviour to gradle?
>
> That would make a lot of sense. In fact, as soon as we support
> arbitrary multi-project layouts this is even required. Right now the
> only way to achieve what you want (i.e. partial build) is to go to the
> subproject folder and execute the build from there. This works only
> because we assume a hierarchical layout and are able to find the root
> by looking for a settings.gradle file. As soon as we can't make any
> assumptions about the layout, you either have to specify the root when
> you are in the subproject or the subproject, if you are in the root.
> The latter is what you want.
>
In the case where a path is provided on the command-line, should we
search for the task in the current project (and its sub-projects) or the
root project (and its sub-projects)?

ie in the water sample, if I am in the krill sub-directory, should I be
able to run gradle :bluewhale:hello? Or just :krill:hello?

Searching in the current project would be consistent with the case where
a name is provided.  Searching in the root project would be consistent
with the behaviour of task dependsOn, and possibly a better fit for
arbitrary multi-project layouts as we don't need to know what the
current project is.


Adam

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

hans_d
Administrator
Hi Adam,

On Jul 21, 2008, at 1:33 PM, Adam Murdoch wrote:

>
>
> Hans Dockter wrote:
>> Hi Adam,
>>
>> On Jul 16, 2008, at 12:32 AM, Adam Murdoch wrote:
>>> More specifically, when a name is provided on the command-line,  
>>> if it starts with ':', it is treated as a task path, and the task  
>>> with that path is executed (along with its dependencies).  
>>> Otherwise, it is treated as it is currently, where all tasks with  
>>> the given name are executed (with dependencies).
>>>
>>> Would it make sense to add this behaviour to gradle?
>>
>> That would make a lot of sense. In fact, as soon as we support  
>> arbitrary multi-project layouts this is even required. Right now  
>> the only way to achieve what you want (i.e. partial build) is to  
>> go to the subproject folder and execute the build from there. This  
>> works only because we assume a hierarchical layout and are able to  
>> find the root by looking for a settings.gradle file. As soon as we  
>> can't make any assumptions about the layout, you either have to  
>> specify the root when you are in the subproject or the subproject,  
>> if you are in the root. The latter is what you want.
>>
> In the case where a path is provided on the command-line, should we  
> search for the task in the current project (and its sub-projects)  
> or the root project (and its sub-projects)?
>
> ie in the water sample, if I am in the krill sub-directory, should  
> I be able to run gradle :bluewhale:hello? Or just :krill:hello?
>
> Searching in the current project would be consistent with the case  
> where a name is provided.  Searching in the root project would be  
> consistent with the behaviour of task dependsOn, and possibly a  
> better fit for arbitrary multi-project layouts as we don't need to  
> know what the current project is.

What about? If we take the future arbitrary layouts into account:
- You can run a gradle build from anywhere if you specify the root  
project and an absolute task name (anywhere is of course also from a  
subproject)

For arbitrary layouts we might do also:
- If you specify a relative pathname in a subproject and specify a  
root project, the task with the relative task is executed against the  
subproject.

For hierarchical layouts:
- If you specify a relative pathname in a subproject we continue to  
do what we do know.
- If you specify a absolute path in a subproject, the same happens as  
if you do it from the root project.

What do you think?

- Hans

>
>
> Adam
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

Adam Murdoch-2
In reply to this post by hans_d


Hans Dockter wrote:

> Hi Adam,
>
> On Jul 17, 2008, at 12:48 AM, Adam Murdoch wrote:
>
>>
>> - How do you decide what counts as the 'core', and therefore needs to
>> be implemented in java, and what does not count as 'core', and
>> therefore can be (must be?) implemented in groovy.
>
> The core is everything except the DSL layer. As we are in the midst of
> a refactoring from Groovy to Java this layer isn't yet architecturally
> as separated as it should be. More or less everything except the
> DefaultProject, DefaultTask and the org.gradle.groovy package should
> be in Java.
>
How about the unit tests for the core? I notice that a lot of the core
unit tests are implemented in groovy.  Is the plan to keep these, or
port them to java? how about new tests? Should they be implemented in
java? groovy? don't care?

Personally, I like the idea that we can implement the unit tests using
groovy to take advantage of the nice scripting stuff it provides.


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

Adam Murdoch-2
In reply to this post by hans_d


Hans Dockter wrote:
Hi Adam,

On Jul 21, 2008, at 1:33 PM, Adam Murdoch wrote:



Hans Dockter wrote:
Hi Adam,

On Jul 16, 2008, at 12:32 AM, Adam Murdoch wrote:
More specifically, when a name is provided on the command-line, if it starts with ':', it is treated as a task path, and the task with that path is executed (along with its dependencies). Otherwise, it is treated as it is currently, where all tasks with the given name are executed (with dependencies).

Would it make sense to add this behaviour to gradle?

That would make a lot of sense. In fact, as soon as we support arbitrary multi-project layouts this is even required. Right now the only way to achieve what you want (i.e. partial build) is to go to the subproject folder and execute the build from there. This works only because we assume a hierarchical layout and are able to find the root by looking for a settings.gradle file. As soon as we can't make any assumptions about the layout, you either have to specify the root when you are in the subproject or the subproject, if you are in the root. The latter is what you want.

In the case where a path is provided on the command-line, should we search for the task in the current project (and its sub-projects) or the root project (and its sub-projects)?

ie in the water sample, if I am in the krill sub-directory, should I be able to run gradle :bluewhale:hello? Or just :krill:hello?

Searching in the current project would be consistent with the case where a name is provided.  Searching in the root project would be consistent with the behaviour of task dependsOn, and possibly a better fit for arbitrary multi-project layouts as we don't need to know what the current project is.

What about? If we take the future arbitrary layouts into account:
- You can run a gradle build from anywhere if you specify the root project and an absolute task name (anywhere is of course also from a subproject)

For arbitrary layouts we might do also:
- If you specify a relative pathname in a subproject and specify a root project, the task with the relative task is executed against the subproject.

For hierarchical layouts:
- If you specify a relative pathname in a subproject we continue to do what we do know.
- If you specify a absolute path in a subproject, the same happens as if you do it from the root project.

What do you think?

I think it sounds good. Just to clarify a few things:

For arbitrary layouts, is the plan to keep the domain model the same as we have, and simply be more flexible about where the artifacts (build.gradle, etc) can be physically located?

That is, will we always end up with a hierarchy of projects, regardless of where those projects are located and how they are configured?  And will we always have a current (sub)project, regardless of how we determine which one it is?

I think there are 3 types of paths which could be specified on the command-line:

- an unqualified path, ie one without a ':' character. eg 'hello'
- a relative path, ie unqualified-path ( ':' unqualified-path )+  eg 'krill:hello'
- an absolute path ie ':' relative-path eg ':krill:hello'

Given the above, we could interpret each type of path as:
- unqualified path: execute all tasks whose path matches current-project:**:path (using an ant style pattern here) where current-project is the path of the current project and path is the value supplied on the command-line
- relative path: execute all tasks whose path matches current-project:path, or maybe current-project:**:path
- absolute path: execute the task whose path matches path

Is this what you meant? (ignoring how we determine the root project, sub-projects and current project)

It might even be interesting to extend this to allow ant-style patterns to be used, eg :krill:* executes all tasks in :krill, :krill:*:hello executes the hello task in the direct subprojects of :krill, etc.


Adam

--------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

hans_d
Administrator
In reply to this post by Adam Murdoch-2
Hi Adam,

On Jul 22, 2008, at 4:46 AM, Adam Murdoch wrote:

>
>
> Hans Dockter wrote:
>> Hi Adam,
>>
>> On Jul 17, 2008, at 12:48 AM, Adam Murdoch wrote:
>>
>>>
>>> - How do you decide what counts as the 'core', and therefore  
>>> needs to be implemented in java, and what does not count as  
>>> 'core', and therefore can be (must be?) implemented in groovy.
>>
>> The core is everything except the DSL layer. As we are in the  
>> midst of a refactoring from Groovy to Java this layer isn't yet  
>> architecturally as separated as it should be. More or less  
>> everything except the DefaultProject, DefaultTask and the  
>> org.gradle.groovy package should be in Java.
>>
> How about the unit tests for the core? I notice that a lot of the  
> core unit tests are implemented in groovy.  Is the plan to keep  
> these, or port them to java? how about new tests? Should they be  
> implemented in java? groovy? don't care?
>
> Personally, I like the idea that we can implement the unit tests  
> using groovy to take advantage of the nice scripting stuff it  
> provides.

I don't feel strong about this. There are two issues I have with  
Groovy unit tests.

- Performance: We have an abstract test case which is extended by all  
of our task tests. I have refactored this test from Groovy to Java  
last week and the execution time of our complete test suite (900  
tests) went down from 30 seconds to 15 seconds on my machine.

- Mocking: The Groovy mocking framework does not work if the class  
under test is written in Java. They want to change this, but that is  
the way it is right now. So for all future unit tests written in  
Groovy we should use JMock. There is a JUnit4GroovyMockery class in  
the test source which I have copied from the Groovy Wiki regarding  
testing. You might search usages of this class on how to use it. You  
can't use JMock the Java way in Groovy code. The JUnit4GroovyMockery  
class is fine for normal use cases. But I could not make it work for  
more complex argument matching like:

context.checking {
        one(myMock).someMethod(with(any(String.class))
}

I have posted this to the Groovy mailing list and they might come up  
with a solution.

I see also the benefits of using Groovy for unit tests. We therefore  
might leave it up to a developer to decide if we are able to solve  
the JMock issue.

- Hans

--
Hans Dockter
Gradle Project lead
http://www.gradle.org





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Executing child project tasks from the command line

Adam Murdoch-2
In reply to this post by Adam Murdoch-2


Adam Murdoch wrote:


Hans Dockter wrote:
Hi Adam,

On Jul 21, 2008, at 1:33 PM, Adam Murdoch wrote:



Hans Dockter wrote:
Hi Adam,

On Jul 16, 2008, at 12:32 AM, Adam Murdoch wrote:
More specifically, when a name is provided on the command-line, if it starts with ':', it is treated as a task path, and the task with that path is executed (along with its dependencies). Otherwise, it is treated as it is currently, where all tasks with the given name are executed (with dependencies).

Would it make sense to add this behaviour to gradle?

That would make a lot of sense. In fact, as soon as we support arbitrary multi-project layouts this is even required. Right now the only way to achieve what you want (i.e. partial build) is to go to the subproject folder and execute the build from there. This works only because we assume a hierarchical layout and are able to find the root by looking for a settings.gradle file. As soon as we can't make any assumptions about the layout, you either have to specify the root when you are in the subproject or the subproject, if you are in the root. The latter is what you want.

In the case where a path is provided on the command-line, should we search for the task in the current project (and its sub-projects) or the root project (and its sub-projects)?

ie in the water sample, if I am in the krill sub-directory, should I be able to run gradle :bluewhale:hello? Or just :krill:hello?

Searching in the current project would be consistent with the case where a name is provided.  Searching in the root project would be consistent with the behaviour of task dependsOn, and possibly a better fit for arbitrary multi-project layouts as we don't need to know what the current project is.

What about? If we take the future arbitrary layouts into account:
- You can run a gradle build from anywhere if you specify the root project and an absolute task name (anywhere is of course also from a subproject)

For arbitrary layouts we might do also:
- If you specify a relative pathname in a subproject and specify a root project, the task with the relative task is executed against the subproject.

For hierarchical layouts:
- If you specify a relative pathname in a subproject we continue to do what we do know.
- If you specify a absolute path in a subproject, the same happens as if you do it from the root project.

What do you think?

I think it sounds good. Just to clarify a few things:

For arbitrary layouts, is the plan to keep the domain model the same as we have, and simply be more flexible about where the artifacts (build.gradle, etc) can be physically located?

That is, will we always end up with a hierarchy of projects, regardless of where those projects are located and how they are configured?  And will we always have a current (sub)project, regardless of how we determine which one it is?

I think there are 3 types of paths which could be specified on the command-line:

- an unqualified path, ie one without a ':' character. eg 'hello'
- a relative path, ie unqualified-path ( ':' unqualified-path )+  eg 'krill:hello'
- an absolute path ie ':' relative-path eg ':krill:hello'

Given the above, we could interpret each type of path as:
- unqualified path: execute all tasks whose path matches current-project:**:path (using an ant style pattern here) where current-project is the path of the current project and path is the value supplied on the command-line
- relative path: execute all tasks whose path matches current-project:path, or maybe current-project:**:path
- absolute path: execute the task whose path matches path

I've created a jira issue for this, and attached a patch which pretty much implements this.

http://jira.codehaus.org/browse/GRADLE-165

I made a few structural changes as part of this patch.  I moved the knowledge of selecting which tasks to execute, project default tasks, and validation out of Build and BuildExecutor behind a new TaskSelector interface and a couple of implementations which Build delegates to.

I also removed the --non-recursive command-line option.


Adam

--------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email