performance

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

performance

Steve Appling
After Hans' dependency resolution performance fix, the huge performance problem
we saw has been fixed (thanks Hans!).  I still find, however, that the overhead
of all the up to date checking is not worth it.  When everything is up to date
and there is nothing to do, running "build -xtest" for our big project in the
0.8 version takes about 1:11 (min:sec) and doing the same thing with the current
trunk version takes 1:24.

Is there a way to disable the up to date checks for particular tasks?  I don't
think it is currently useful for compileJava or Copy type of tasks and would
like to try disabling it there.

--
Steve Appling
Automated Logic Research Team

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

hans_d
Administrator

On Nov 20, 2009, at 8:05 PM, Steve Appling wrote:

> After Hans' dependency resolution performance fix, the huge performance problem we saw has been fixed (thanks Hans!).  I still find, however, that the overhead of all the up to date checking is not worth it.  When everything is up to date and there is nothing to do, running "build -xtest" for our big project in the 0.8 version takes about 1:11 (min:sec) and doing the same thing with the current trunk version takes 1:24.

In my test project I had the following profiling results. For a build were nothing needed to be done, 40 percent of the time consumed Ivy for coming up with the dependencies. The state change checking needs the classpath to do comparisons. 10 percent of the time was consumed by our Ivy result translation (we might be able to improve this). This test project did not have many source files. It will be interesting to see whether with a larger codebase source change detection will play an important role here.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org

>
> Is there a way to disable the up to date checks for particular tasks?  I don't think it is currently useful for compileJava or Copy type of tasks and would like to try disabling it there.


>
> --
> Steve Appling
> Automated Logic Research Team
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

Steve Appling
Do you want another profile snapshot from us now that your fix has been applied?
  If so, is sampling ok?

Hans Dockter wrote:

> On Nov 20, 2009, at 8:05 PM, Steve Appling wrote:
>
>> After Hans' dependency resolution performance fix, the huge performance problem we saw has been fixed (thanks Hans!).  I still find, however, that the overhead of all the up to date checking is not worth it.  When everything is up to date and there is nothing to do, running "build -xtest" for our big project in the 0.8 version takes about 1:11 (min:sec) and doing the same thing with the current trunk version takes 1:24.
>
> In my test project I had the following profiling results. For a build were nothing needed to be done, 40 percent of the time consumed Ivy for coming up with the dependencies. The state change checking needs the classpath to do comparisons. 10 percent of the time was consumed by our Ivy result translation (we might be able to improve this). This test project did not have many source files. It will be interesting to see whether with a larger codebase source change detection will play an important role here.
>
> - Hans
>
> --
> Hans Dockter
> Gradle Project Manager
> http://www.gradle.org
>
>> Is there a way to disable the up to date checks for particular tasks?  I don't think it is currently useful for compileJava or Copy type of tasks and would like to try disabling it there.
>
>
>> --
>> Steve Appling
>> Automated Logic Research Team
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>   http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>

--
Steve Appling
Automated Logic Research Team

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

Adam Murdoch-2
In reply to this post by Steve Appling


Steve Appling wrote:

> After Hans' dependency resolution performance fix, the huge
> performance problem we saw has been fixed (thanks Hans!).  I still
> find, however, that the overhead of all the up to date checking is not
> worth it.  When everything is up to date and there is nothing to do,
> running "build -xtest" for our big project in the 0.8 version takes
> about 1:11 (min:sec) and doing the same thing with the current trunk
> version takes 1:24.
>
> Is there a way to disable the up to date checks for particular tasks?  
> I don't think it is currently useful for compileJava or Copy type of
> tasks and would like to try disabling it there.
>

How about we make the checking faster, instead? The new stuff has not
been heavily profiled or optimised. I was hoping to get back into making
this faster next week. I'm pretty confident we can get close for 'build
-x test'. Of course, plain 'build' should already be faster.

Of course, your numbers don't really say anything more than something is
slower in trunk. It could be anything, so let's measure before we
optimise. Any chance you could send me a profiling snapshot of the above
build. Or if not, I can add some logging and we can see where the time
is being taken.

--
Adam Murdoch
Gradle Developer
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

hans_d
Administrator
In reply to this post by Steve Appling

On Nov 20, 2009, at 8:47 PM, Steve Appling wrote:

> Do you want another profile snapshot from us now that your fix has been applied?  If so, is sampling ok?

As Adam said, that would be good to have. Sampling would be OK I think.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org

>
> Hans Dockter wrote:
>> On Nov 20, 2009, at 8:05 PM, Steve Appling wrote:
>>> After Hans' dependency resolution performance fix, the huge performance problem we saw has been fixed (thanks Hans!).  I still find, however, that the overhead of all the up to date checking is not worth it.  When everything is up to date and there is nothing to do, running "build -xtest" for our big project in the 0.8 version takes about 1:11 (min:sec) and doing the same thing with the current trunk version takes 1:24.
>> In my test project I had the following profiling results. For a build were nothing needed to be done, 40 percent of the time consumed Ivy for coming up with the dependencies. The state change checking needs the classpath to do comparisons. 10 percent of the time was consumed by our Ivy result translation (we might be able to improve this). This test project did not have many source files. It will be interesting to see whether with a larger codebase source change detection will play an important role here. - Hans
>> --
>> Hans Dockter
>> Gradle Project Manager
>> http://www.gradle.org
>>> Is there a way to disable the up to date checks for particular tasks?  I don't think it is currently useful for compileJava or Copy type of tasks and would like to try disabling it there.
>>> --
>>> Steve Appling
>>> Automated Logic Research Team
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>  http://xircles.codehaus.org/manage_email
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>    http://xircles.codehaus.org/manage_email
>
> --
> Steve Appling
> Automated Logic Research Team
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

Steve Appling
In reply to this post by Adam Murdoch-2
I'll send a snapshot to both of you off of the list.  It looks to me like
40% of the time is spent in o.g.a.i.changedetection.CachingHasher.hash.  That's
why I was thinking it was the up to date checks.  How about an option to make
this just based on timestamp and length instead of hashing the contents?

Adam Murdoch wrote:

>
>
> Steve Appling wrote:
>> After Hans' dependency resolution performance fix, the huge
>> performance problem we saw has been fixed (thanks Hans!).  I still
>> find, however, that the overhead of all the up to date checking is not
>> worth it.  When everything is up to date and there is nothing to do,
>> running "build -xtest" for our big project in the 0.8 version takes
>> about 1:11 (min:sec) and doing the same thing with the current trunk
>> version takes 1:24.
>>
>> Is there a way to disable the up to date checks for particular tasks?  
>> I don't think it is currently useful for compileJava or Copy type of
>> tasks and would like to try disabling it there.
>>
>
> How about we make the checking faster, instead? The new stuff has not
> been heavily profiled or optimised. I was hoping to get back into making
> this faster next week. I'm pretty confident we can get close for 'build
> -x test'. Of course, plain 'build' should already be faster.
>
> Of course, your numbers don't really say anything more than something is
> slower in trunk. It could be anything, so let's measure before we
> optimise. Any chance you could send me a profiling snapshot of the above
> build. Or if not, I can add some logging and we can see where the time
> is being taken.
>

--
Steve Appling
Automated Logic Research Team

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: performance

Adam Murdoch-2


Steve Appling wrote:
> I'll send a snapshot to both of you off of the list.  It looks to me like
> 40% of the time is spent in
> o.g.a.i.changedetection.CachingHasher.hash.  That's why I was thinking
> it was the up to date checks.  How about an option to make this just
> based on timestamp and length instead of hashing the contents?

It should calculate a hash only when timestamp or length changes. Of
course, it may be broken ...

>
> Adam Murdoch wrote:
>>
>>
>> Steve Appling wrote:
>>> After Hans' dependency resolution performance fix, the huge
>>> performance problem we saw has been fixed (thanks Hans!).  I still
>>> find, however, that the overhead of all the up to date checking is
>>> not worth it.  When everything is up to date and there is nothing to
>>> do, running "build -xtest" for our big project in the 0.8 version
>>> takes about 1:11 (min:sec) and doing the same thing with the current
>>> trunk version takes 1:24.
>>>
>>> Is there a way to disable the up to date checks for particular
>>> tasks?  I don't think it is currently useful for compileJava or Copy
>>> type of tasks and would like to try disabling it there.
>>>
>>
>> How about we make the checking faster, instead? The new stuff has not
>> been heavily profiled or optimised. I was hoping to get back into
>> making this faster next week. I'm pretty confident we can get close
>> for 'build -x test'. Of course, plain 'build' should already be faster.
>>
>> Of course, your numbers don't really say anything more than something
>> is slower in trunk. It could be anything, so let's measure before we
>> optimise. Any chance you could send me a profiling snapshot of the
>> above build. Or if not, I can add some logging and we can see where
>> the time is being taken.
>>
>

--
Adam Murdoch
Gradle Developer
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email