task artifact cache storing absolute paths, preventing CI workspace reuse.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

task artifact cache storing absolute paths, preventing CI workspace reuse.

Luke Daley-2
Hi,

http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location

I imagine that guaranteeing that the task artifact cache is completely
portable would be quite difficult, but could we make all paths relative?
This might be good enough portability in some use cases (though
admittedly I don't know much about that machinery so that could be a
grossly inaccurate statement).

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: task artifact cache storing absolute paths, preventing CI workspace reuse.

Radim Kubacki
On Thu, Feb 13, 2014 at 2:14 AM, Luke Daley <[hidden email]> wrote:
Hi,

http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location

I imagine that guaranteeing that the task artifact cache is completely portable would be quite difficult, but could we make all paths relative? This might be good enough portability in some use cases (though admittedly I don't know much about that machinery so that could be a grossly inaccurate statement).

I agree that we should make paths relative where possible. Relative to project dir or gradle_user_home or gradle installation. Remaining parts are external dependencies. Some of them are expected (system header files, JDK installation) the rest should often be avoided.

BTW the motivation for the topic comes from that problem with build pipelines on CI server: how to reuse results from one build job in another one. Making paths relative can help in some cases. If will not help if I build & unit test my app on one OS and then start functional tests on Linux/Win/Mac and avoid rebuilds. One idea is to play with dependencies - my local run of funcTest can depend on source build (main sourceSet) and CI server will pass a flag to change it to a dependency on an artifact built by previous job. Does it makes sense? Is it something what we want to promote?
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: task artifact cache storing absolute paths, preventing CI workspace reuse.

Adam Murdoch
In reply to this post by Luke Daley-2

On 12 Feb 2014, at 5:14 pm, Luke Daley <[hidden email]> wrote:

Hi,

http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location

I imagine that guaranteeing that the task artifact cache is completely portable would be quite difficult, but could we make all paths relative? This might be good enough portability in some use cases (though admittedly I don't know much about that machinery so that could be a grossly inaccurate statement).

We can, yes. There are a few cases where it won’t work:

- There are absolute paths in the task’s input property values (eg Test and BuildDashboard)
- The task maintains any kind of incremental state that the itself (eg the incremental compile tasks).
- The task output is not portable (eg absolute paths baked into native binaries at link time).

I’m sure there are more. The first two are probably ok to ignore, as they just mean we end up doing more work than we should in the new workspace. The third is potentially quite nasty, because we would skip work that we should be doing, leaving the output pointing to files in the old location rather than the copies in the new location. This is not good from an incremental build point of view.

The problem here is that I don’t really want to guarantee anything about portability, as I think there are probably better ways to solve this use case. We can probably make it good enough, but we’d need to know whether the task output is portable or not.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: task artifact cache storing absolute paths, preventing CI workspace reuse.

Adam Murdoch
In reply to this post by Radim Kubacki

On 13 Feb 2014, at 1:16 am, Radim Kubacki <[hidden email]> wrote:

On Thu, Feb 13, 2014 at 2:14 AM, Luke Daley <[hidden email]> wrote:
Hi,

http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location

I imagine that guaranteeing that the task artifact cache is completely portable would be quite difficult, but could we make all paths relative? This might be good enough portability in some use cases (though admittedly I don't know much about that machinery so that could be a grossly inaccurate statement).

I agree that we should make paths relative where possible. Relative to project dir or gradle_user_home or gradle installation. Remaining parts are external dependencies. Some of them are expected (system header files, JDK installation) the rest should often be avoided.

BTW the motivation for the topic comes from that problem with build pipelines on CI server: how to reuse results from one build job in another one. Making paths relative can help in some cases. If will not help if I build & unit test my app on one OS and then start functional tests on Linux/Win/Mac and avoid rebuilds. One idea is to play with dependencies - my local run of funcTest can depend on source build (main sourceSet) and CI server will pass a flag to change it to a dependency on an artifact built by previous job. Does it makes sense? Is it something what we want to promote?

Yes, this is how I would like to solve the use case. It’s kind of awkward at the moment, though. This is something we will address as part of the component dependency management stuff we’re doing for native and Android components (and JVM components too).


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: task artifact cache storing absolute paths, preventing CI workspace reuse.

Luke Daley-2
In reply to this post by Adam Murdoch
So it seems the best thing would be to post back to the forum user that
this will in effect never be “supported”, but if they want to live
dangerously then we would accept a PR to relativize the particular paths
in question.

Agree?

> Adam Murdoch <mailto:[hidden email]>
> 14 February 2014 1:36 am
>
> We can, yes. There are a few cases where it won’t work:
>
> - There are absolute paths in the task’s input property values (eg
> Test and BuildDashboard)
> - The task maintains any kind of incremental state that the itself (eg
> the incremental compile tasks).
> - The task output is not portable (eg absolute paths baked into native
> binaries at link time).
>
> I’m sure there are more. The first two are probably ok to ignore, as
> they just mean we end up doing more work than we should in the new
> workspace. The third is potentially quite nasty, because we would skip
> work that we should be doing, leaving the output pointing to files in
> the old location rather than the copies in the new location. This is
> not good from an incremental build point of view.
>
> The problem here is that I don’t really want to guarantee anything
> about portability, as I think there are probably better ways to solve
> this use case. We can probably make it good enough, but we’d need to
> know whether the task output is portable or not.
>
>
> --
> Adam Murdoch
> Gradle Co-founder
> http://www.gradle.org
> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
> http://www.gradleware.com
>
>
>
>
> Luke Daley <mailto:[hidden email]>
> 13 February 2014 11:14 am
> Hi,
>
> http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location 
>
>
> I imagine that guaranteeing that the task artifact cache is completely
> portable would be quite difficult, but could we make all paths
> relative? This might be good enough portability in some use cases
> (though admittedly I don't know much about that machinery so that
> could be a grossly inaccurate statement).

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: task artifact cache storing absolute paths, preventing CI workspace reuse.

Adam Murdoch

On 16 Feb 2014, at 4:31 pm, Luke Daley <[hidden email]> wrote:

So it seems the best thing would be to post back to the forum user that this will in effect never be “supported”, but if they want to live dangerously then we would accept a PR to relativize the particular paths in question.

Agree?

Pretty much, except for the “never” bit. I think we’ll get to this point eventually, but it’s a long way off.


Adam Murdoch <[hidden email]>
14 February 2014 1:36 am

We can, yes. There are a few cases where it won’t work:

- There are absolute paths in the task’s input property values (eg Test and BuildDashboard)
- The task maintains any kind of incremental state that the itself (eg the incremental compile tasks).
- The task output is not portable (eg absolute paths baked into native binaries at link time).

I’m sure there are more. The first two are probably ok to ignore, as they just mean we end up doing more work than we should in the new workspace. The third is potentially quite nasty, because we would skip work that we should be doing, leaving the output pointing to files in the old location rather than the copies in the new location. This is not good from an incremental build point of view.

The problem here is that I don’t really want to guarantee anything about portability, as I think there are probably better ways to solve this use case. We can probably make it good enough, but we’d need to know whether the task output is portable or not.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com




Luke Daley <[hidden email]>
13 February 2014 11:14 am
Hi,

http://forums.gradle.org/gradle/topics/incremental_build_state_should_be_decoupled_from_workspace_location 

I imagine that guaranteeing that the task artifact cache is completely portable would be quite difficult, but could we make all paths relative? This might be good enough portability in some use cases (though admittedly I don't know much about that machinery so that could be a grossly inaccurate statement).

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com