Standardising the JUnit XML test report format

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Standardising the JUnit XML test report format

Lee Symes
Hi All,

I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post

So when a test is skipped in Gradle it produces(for example):
<testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
  <ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

However when Ant runs the same test(changes highlighted):
<testsuite name="com.foo.FooTest" tests="1"skipped="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
  <testcase name="some skipped test" classname="com.foo.FooTest" time="0.01">
    <skipped/>
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase.

The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format.

I am writing a fix for this so is there anything that I have missed or overlooked in the above.

I hope that I can help out,

Lee 
Reply | Threaded
Open this post in threaded view
|

Re: Standardising the JUnit XML test report format

Luke Daley-2
We've been pretty conservative about making changes to this in the past.

We did make a change a little while ago to add the ability to track output per test case, and we added an opt in switch for it (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)). In this case though, I think we should just make the change after testing for compatibility with different tools. The chances that something has specific handling for Gradle's format is low.

Can anyone recall why we deviate from Ant and Maven here?

On 9 Feb 2014, at 5:32 pm, Lee Symes <[hidden email]> wrote:

> Hi All,
>
> I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post.
>
> So when a test is skipped in Gradle it produces(for example):
> <testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
>   <ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/>
>   <system-out><![CDATA[]]></system-out>
>   <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> However when Ant runs the same test(changes highlighted):
> <testsuite name="com.foo.FooTest" tests="1"skipped="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
>   <testcase name="some skipped test" classname="com.foo.FooTest" time="0.01">
>     <skipped/>
>   </testcase>
>   <system-out><![CDATA[]]></system-out>
>   <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase.
>
> The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format.
>
> I am writing a fix for this so is there anything that I have missed or overlooked in the above.
>
> I hope that I can help out,
>
> Lee

--
Luke Daley
Principal Engineer, Gradleware
http://gradleware.com


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Standardising the JUnit XML test report format

Lee Symes
Hi again,

I've committed my code into my fork of Gradle(https://github.com/leesdolphin/gradle/compare). In my searches though I have noticed that there was a pull request(https://github.com/gradle/gradle/pull/183) to make effectively the same changes. This pull request was shelved for backwards compatibility reasons.

I have added a skipped counter to the JUnit TestClassResult. Also changed the generation code to:
  • Add the skipped annotation.
  • Output the ignored test case element name as testcase and added an empty sub element <skipped/>

I have also changed the test specs & helper objects to include the changes to annotations and element naming.

I'll submit a pull request tomorrow once I have submitted my CLA.

Yours,

Lee



On 10 February 2014 07:34, Luke Daley <[hidden email]> wrote:
We've been pretty conservative about making changes to this in the past.

We did make a change a little while ago to add the ability to track output per test case, and we added an opt in switch for it (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)). In this case though, I think we should just make the change after testing for compatibility with different tools. The chances that something has specific handling for Gradle's format is low.

Can anyone recall why we deviate from Ant and Maven here?

On 9 Feb 2014, at 5:32 pm, Lee Symes <[hidden email]> wrote:

> Hi All,
>
> I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post.
>
> So when a test is skipped in Gradle it produces(for example):
> <testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
>   <ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/>
>   <system-out><![CDATA[]]></system-out>
>   <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> However when Ant runs the same test(changes highlighted):
> <testsuite name="com.foo.FooTest" tests="1"skipped="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
>   <testcase name="some skipped test" classname="com.foo.FooTest" time="0.01">
>     <skipped/>
>   </testcase>
>   <system-out><![CDATA[]]></system-out>
>   <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase.
>
> The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format.
>
> I am writing a fix for this so is there anything that I have missed or overlooked in the above.
>
> I hope that I can help out,
>
> Lee

--
Luke Daley
Principal Engineer, Gradleware
http://gradleware.com


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Standardising the JUnit XML test report format

Luke Daley-2
Hi Lee,

Did you submit this pull request? We'd like to get this into 1.12.

> Lee Symes <mailto:[hidden email]>
> 12 February 2014 9:29 pm
> Hi again,
>
> I've committed my code into my fork of
> Gradle(https://github.com/leesdolphin/gradle/compare). In my searches
> though I have noticed that there was a pull
> request(https://github.com/gradle/gradle/pull/183) to make effectively
> the same changes. This pull request was shelved for backwards
> compatibility reasons.
>
> I have added a skipped counter to the JUnit TestClassResult. Also
> changed the generation code to:
>
>   * Add the skipped annotation.
>   * Output the ignored test case element name as testcase and added an
>     empty sub element <skipped/>
>
> I have also changed the test specs & helper objects to include the
> changes to annotations and element naming.
>
> I'll submit a pull request tomorrow once I have submitted my CLA.
>
> Yours,
>
> Lee
>
>
>
>
> Luke Daley <mailto:[hidden email]>
> 10 February 2014 7:34 am
> We've been pretty conservative about making changes to this in the past.
>
> We did make a change a little while ago to add the ability to track
> output per test case, and we added an opt in switch for it
> (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)).
> In this case though, I think we should just make the change after
> testing for compatibility with different tools. The chances that
> something has specific handling for Gradle's format is low.
>
> Can anyone recall why we deviate from Ant and Maven here?
>
>
> Lee Symes <mailto:[hidden email]>
> 9 February 2014 5:32 pm
> Hi All,
>
> I have noticed that the JUnit XML reports generated by Gradle are not
> being correctly parsed by Jenkins and other applications when a test
> is skipped. This is in relation to
> http://issues.gradle.org/browse/GRADLE-2731. From what I can find out
> there is no 'standard' format that these reports produce, however Ant
> and Maven(at least) have standardised on the format described in the
> issue and the linked StackOverflow post
> <http://stackoverflow.com/questions/4922867/junit-xml-format-specification-that-hudson-supports>.
>
>
> So when a test is skipped in Gradle it produces(for example):
> <testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0"
> timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
> <ignored-testcase name="some skipped test" classname="com.foo.FooTest"
> time="0.01"/>
> <system-out><![CDATA[]]></system-out>
> <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> However when Ant runs the same test(changes highlighted):
> <testsuite name="com.foo.FooTest" tests="1"*skipped="1"* failures="0"
> errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost"
> time="0.045">
> <*testcase* name="some skipped test" classname="com.foo.FooTest"
> time="0.01">
> *<skipped/>*
> </testcase>
> <system-out><![CDATA[]]></system-out>
> <system-err><![CDATA[]]></system-err>
> </testsuite>
>
> Ant generates a normal testcase tag and adds a child element; and Ant
> also generates a skipped attribute. This is compared to the
> ignored-testcase.
>
> The only concern that I have with changing this output would be any
> tool that has changed to only support the Gradle format.
>
> I am writing a fix for this so is there anything that I have missed or
> overlooked in the above.
>
> I hope that I can help out,
>
> Lee

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Standardising the JUnit XML test report format

Lee Symes
Hi Luke,

My apologies; It had slipped my mind! I have sent the CLA and added a pull request(https://github.com/gradle/gradle/pull/247).

Yours,

Lee


On 17 February 2014 11:34, Luke Daley <[hidden email]> wrote:
Hi Lee,

Did you submit this pull request? We'd like to get this into 1.12.

Lee Symes <mailto:[hidden email]>
12 February 2014 9:29 pm

Hi again,

I've committed my code into my fork of Gradle(https://github.com/leesdolphin/gradle/compare). In my searches though I have noticed that there was a pull request(https://github.com/gradle/gradle/pull/183) to make effectively the same changes. This pull request was shelved for backwards compatibility reasons.

I have added a skipped counter to the JUnit TestClassResult. Also changed the generation code to:

  * Add the skipped annotation.
  * Output the ignored test case element name as testcase and added an

    empty sub element <skipped/>

I have also changed the test specs & helper objects to include the changes to annotations and element naming.

I'll submit a pull request tomorrow once I have submitted my CLA.

Yours,

Lee




Luke Daley <mailto:[hidden email]>
10 February 2014 7:34 am

We've been pretty conservative about making changes to this in the past.

We did make a change a little while ago to add the ability to track output per test case, and we added an opt in switch for it (http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/testing/JUnitXmlReport.html#setOutputPerTestCase(boolean)). In this case though, I think we should just make the change after testing for compatibility with different tools. The chances that something has specific handling for Gradle's format is low.

Can anyone recall why we deviate from Ant and Maven here?


Lee Symes <mailto:[hidden email]>
9 February 2014 5:32 pm
Hi All,

I have noticed that the JUnit XML reports generated by Gradle are not being correctly parsed by Jenkins and other applications when a test is skipped. This is in relation to http://issues.gradle.org/browse/GRADLE-2731. From what I can find out there is no 'standard' format that these reports produce, however Ant and Maven(at least) have standardised on the format described in the issue and the linked StackOverflow post <http://stackoverflow.com/questions/4922867/junit-xml-format-specification-that-hudson-supports>.

So when a test is skipped in Gradle it produces(for example):
<testsuite name="com.foo.FooTest" tests="1" failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
<ignored-testcase name="some skipped test" classname="com.foo.FooTest" time="0.01"/>
<system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>

However when Ant runs the same test(changes highlighted):
<testsuite name="com.foo.FooTest" tests="1"*skipped="1"* failures="0" errors="0" timestamp="2012-11-19T17:09:28" hostname="localhost" time="0.045">
<*testcase* name="some skipped test" classname="com.foo.FooTest" time="0.01">
*<skipped/>*

</testcase>
<system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>

Ant generates a normal testcase tag and adds a child element; and Ant also generates a skipped attribute. This is compared to the ignored-testcase.

The only concern that I have with changing this output would be any tool that has changed to only support the Gradle format.

I am writing a fix for this so is there anything that I have missed or overlooked in the above.

I hope that I can help out,

Lee

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email