TestReport aggregation

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

TestReport aggregation

Radim Kubacki
When I was modifying test aggregation implementation (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8) I found it hard to make it do what I expect: assume I have a set of tests for my project and want to run them several times. Each time it will be run with some special set up (against different DBs, servers, with a different configuration ...). And I want to run them all and see the report rather than stop at first failed test task. 

The problem is that the build fails with first failed test task and the report is not run or I will mark test tasks to ignore failures to get the report and the build will always succeed. I thought that 
1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)
2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)

Opinions?

-Radim
Reply | Threaded
Open this post in threaded view
|

Re: TestReport aggregation

Luke Daley-2


> Radim Kubacki <mailto:[hidden email]>
> 4 March 2014 3:29 am
> When I was modifying test aggregation implementation
> (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8)
> I found it hard to make it do what I expect: assume I have a set of
> tests for my project and want to run them several times. Each time it
> will be run with some special set up (against different DBs, servers,
> with a different configuration ...). And I want to run them all and
> see the report rather than stop at first failed test task.
>
> The problem is that the build fails with first failed test task and
> the report is not run or I will mark test tasks to ignore failures to
> get the report and the build will always succeed. I thought that
> 1. report task should be used as finalizer task (TestReport.reportOn
> should create finalizing dependency)
I'm stunned it doesn't.
> 2. TestReport should emit the message 'There were failing tests. See
> the report at: ...' too if there are failures and set the build result
> accordingly (can be option on that task)
Not so sure that this is the right way.
> Opinions?
>
> -Radim

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: TestReport aggregation

Szczepan Faber-2
Down the road, we should make it dead easy to configure a multiproject build in a way it runs tests from all subprojects before failing. Perhaps it should even be the default for a multiproject build. --continue flag does help to achieve it in some ways and perhaps is good enough for now.

Cheers!


On Tue, Mar 4, 2014 at 12:28 AM, Luke Daley <[hidden email]> wrote:


Radim Kubacki <mailto:[hidden email]>
4 March 2014 3:29 am

When I was modifying test aggregation implementation (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8) I found it hard to make it do what I expect: assume I have a set of tests for my project and want to run them several times. Each time it will be run with some special set up (against different DBs, servers, with a different configuration ...). And I want to run them all and see the report rather than stop at first failed test task.

The problem is that the build fails with first failed test task and the report is not run or I will mark test tasks to ignore failures to get the report and the build will always succeed. I thought that
1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)
I'm stunned it doesn't.

2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)
Not so sure that this is the right way.
Opinions?

-Radim

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
Szczepan Faber
Principal engineer@gradle; Founder@mockito
Reply | Threaded
Open this post in threaded view
|

Re: TestReport aggregation

Radim Kubacki
On Tue, Mar 4, 2014 at 10:01 AM, Szczepan Faber <[hidden email]> wrote:
Down the road, we should make it dead easy to configure a multiproject build in a way it runs tests from all subprojects before failing. Perhaps it should even be the default for a multiproject build. --continue flag does help to achieve it in some ways and perhaps is good enough for now.

I agree that --continue helps. Yet it again falls short and will stop at those test task before proceeding to TestReport. TestReport doesn't declare the dependency directly. It sets its input files to binResultsDir content of tasks it reportsOn. 

-Radim


Cheers!


On Tue, Mar 4, 2014 at 12:28 AM, Luke Daley <[hidden email]> wrote:


Radim Kubacki <mailto:[hidden email]>
4 March 2014 3:29 am

When I was modifying test aggregation implementation (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8) I found it hard to make it do what I expect: assume I have a set of tests for my project and want to run them several times. Each time it will be run with some special set up (against different DBs, servers, with a different configuration ...). And I want to run them all and see the report rather than stop at first failed test task.

The problem is that the build fails with first failed test task and the report is not run or I will mark test tasks to ignore failures to get the report and the build will always succeed. I thought that
1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)
I'm stunned it doesn't.

2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)
Not so sure that this is the right way.
Opinions?

-Radim

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
Szczepan Faber
Principal engineer@gradle; Founder@mockito

Reply | Threaded
Open this post in threaded view
|

Re: TestReport aggregation

Perryn Fowler
Yeah this is exactly what I was talking about in the ' Dependency on task that is finalised by task' thread on this list.

I am now convinced that neither 'dependsOn' nor 'finalizedBy' are the right relationship between Test and TestReport

I think that to fix the fundamental problem we need to model the difference between 'I could not run the tests' and 'I ran the tests successfully, but they failed'. This actually applies for any task that performs some kind of check.

In the meantime though:

>1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)

 I think that TestReport.reportOn setting up a finalizedBy relationship is better than what it does now.

> 2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)

 My worry with this would be that any other task that depends on the Test task would now run if the tests fail - Do you still need to do this if you have done 1?

Pez



On Tue, Mar 4, 2014 at 8:19 PM, Radim Kubacki <[hidden email]> wrote:
On Tue, Mar 4, 2014 at 10:01 AM, Szczepan Faber <[hidden email]> wrote:
Down the road, we should make it dead easy to configure a multiproject build in a way it runs tests from all subprojects before failing. Perhaps it should even be the default for a multiproject build. --continue flag does help to achieve it in some ways and perhaps is good enough for now.

I agree that --continue helps. Yet it again falls short and will stop at those test task before proceeding to TestReport. TestReport doesn't declare the dependency directly. It sets its input files to binResultsDir content of tasks it reportsOn. 

-Radim


Cheers!


On Tue, Mar 4, 2014 at 12:28 AM, Luke Daley <[hidden email]> wrote:


Radim Kubacki <mailto:[hidden email]>
4 March 2014 3:29 am

When I was modifying test aggregation implementation (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8) I found it hard to make it do what I expect: assume I have a set of tests for my project and want to run them several times. Each time it will be run with some special set up (against different DBs, servers, with a different configuration ...). And I want to run them all and see the report rather than stop at first failed test task.

The problem is that the build fails with first failed test task and the report is not run or I will mark test tasks to ignore failures to get the report and the build will always succeed. I thought that
1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)
I'm stunned it doesn't.

2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)
Not so sure that this is the right way.
Opinions?

-Radim

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
Szczepan Faber
Principal engineer@gradle; Founder@mockito


Reply | Threaded
Open this post in threaded view
|

Re: TestReport aggregation

Radim Kubacki
On Wed, Mar 5, 2014 at 3:39 AM, Perryn Fowler <[hidden email]> wrote:
Yeah this is exactly what I was talking about in the ' Dependency on task that is finalised by task' thread on this list.

I am now convinced that neither 'dependsOn' nor 'finalizedBy' are the right relationship between Test and TestReport

I think that to fix the fundamental problem we need to model the difference between 'I could not run the tests' and 'I ran the tests successfully, but they failed'. This actually applies for any task that performs some kind of check.

In the meantime though:

>1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)

 I think that TestReport.reportOn setting up a finalizedBy relationship is better than what it does now.

Seems we have an agreement here.
 
> 2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)

 My worry with this would be that any other task that depends on the Test task would now run if the tests fail - Do you still need to do this if you have done 1?

Hopefully 1 will make it acceptable because as you wrote using ignoreFailures calls for troubles. I haven't tried it yet.
 
Pez



On Tue, Mar 4, 2014 at 8:19 PM, Radim Kubacki <[hidden email]> wrote:
On Tue, Mar 4, 2014 at 10:01 AM, Szczepan Faber <[hidden email]> wrote:
Down the road, we should make it dead easy to configure a multiproject build in a way it runs tests from all subprojects before failing. Perhaps it should even be the default for a multiproject build. --continue flag does help to achieve it in some ways and perhaps is good enough for now.

I agree that --continue helps. Yet it again falls short and will stop at those test task before proceeding to TestReport. TestReport doesn't declare the dependency directly. It sets its input files to binResultsDir content of tasks it reportsOn. 

-Radim


Cheers!


On Tue, Mar 4, 2014 at 12:28 AM, Luke Daley <[hidden email]> wrote:


Radim Kubacki <mailto:[hidden email]>
4 March 2014 3:29 am

When I was modifying test aggregation implementation (https://github.com/gradle/gradle/commit/985f6fa6c137a9903e764fe15818242972c24bc8) I found it hard to make it do what I expect: assume I have a set of tests for my project and want to run them several times. Each time it will be run with some special set up (against different DBs, servers, with a different configuration ...). And I want to run them all and see the report rather than stop at first failed test task.

The problem is that the build fails with first failed test task and the report is not run or I will mark test tasks to ignore failures to get the report and the build will always succeed. I thought that
1. report task should be used as finalizer task (TestReport.reportOn should create finalizing dependency)
I'm stunned it doesn't.

2. TestReport should emit the message 'There were failing tests. See the report at: ...' too if there are failures and set the build result accordingly (can be option on that task)
Not so sure that this is the right way.
Opinions?

-Radim

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
Szczepan Faber
Principal engineer@gradle; Founder@mockito