Dealing with thread count in parallelization

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Dealing with thread count in parallelization

Xavier Ducrohet
Gradle has an option to choose the number of thread (--parallel-thread). I'm guessing that this is used when building different projects in parallel.

In the long term, we're going to have 3 types of parallelizations:
- project level
- task level inside a single project
- the task itself (I'm already doing this in a few task in the Android plugin).

Right now each of my task creates a new thread pool based on the number of cores available which is not great on its own and definitively doesn't work well when building projects in parallel.

Have you thought about having a global thread pool used by all three types? This would allow us to both avoid creating new threads in every tasks but also control better the number of threads running concurrently.

thanks
Xav
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with thread count in parallelization

Adam Murdoch

On 3 Jan 2014, at 8:40 am, Xavier Ducrohet <[hidden email]> wrote:

Gradle has an option to choose the number of thread (--parallel-thread). I'm guessing that this is used when building different projects in parallel.

In the long term, we're going to have 3 types of parallelizations:
- project level
- task level inside a single project
- the task itself (I'm already doing this in a few task in the Android plugin).

Right now each of my task creates a new thread pool based on the number of cores available which is not great on its own and definitively doesn't work well when building projects in parallel.

Have you thought about having a global thread pool used by all three types? This would allow us to both avoid creating new threads in every tasks but also control better the number of threads running concurrently.

Yes, absolutely. We also have the problem with parallel test execution and —parallel, and will run into it when we introduce parallel compilation for the native languages.

The idea is to have an API that tasks can use to spawn pieces of work that can be run concurrently, and Gradle can take care of assigning work to threads. Possibly this might just be something simple like a java.util.concurrent.Executor subtype, or we might make it more structured so that each piece of work that a task spawns can also have its own dependencies, inputs and outputs, incremental state and so on, as a kind of light-weight tasks.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com



Reply | Threaded
Open this post in threaded view
|

Re: Dealing with thread count in parallelization

Xavier Ducrohet
Excellent.

Executor works well for us since we are still trying to provide Gradle as a front to a builder library that is build-system agnostic. We currently use executors in various places to parallelize tasks.


On Sun, Jan 5, 2014 at 12:59 AM, Adam Murdoch <[hidden email]> wrote:

On 3 Jan 2014, at 8:40 am, Xavier Ducrohet <[hidden email]> wrote:

Gradle has an option to choose the number of thread (--parallel-thread). I'm guessing that this is used when building different projects in parallel.

In the long term, we're going to have 3 types of parallelizations:
- project level
- task level inside a single project
- the task itself (I'm already doing this in a few task in the Android plugin).

Right now each of my task creates a new thread pool based on the number of cores available which is not great on its own and definitively doesn't work well when building projects in parallel.

Have you thought about having a global thread pool used by all three types? This would allow us to both avoid creating new threads in every tasks but also control better the number of threads running concurrently.

Yes, absolutely. We also have the problem with parallel test execution and —parallel, and will run into it when we introduce parallel compilation for the native languages.

The idea is to have an API that tasks can use to spawn pieces of work that can be run concurrently, and Gradle can take care of assigning work to threads. Possibly this might just be something simple like a java.util.concurrent.Executor subtype, or we might make it more structured so that each piece of work that a task spawns can also have its own dependencies, inputs and outputs, incremental state and so on, as a kind of light-weight tasks.


--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com